org.apache.hadoop.hbase.regionserver

Class HRegionFileSystem

java.lang.Object

org.apache.hadoop.hbase.regionserver.HRegionFileSystem

Direct Known Subclasses:

HRegionWALFileSystem

@InterfaceAudience.Private
public class HRegionFileSystem
extends Object

View to an on-disk Region. Provides the set of methods necessary to interact with the on-disk region data.

Field Summary

Fields

Modifier and Type	Field and Description
static String	REGION_INFO_FILE Name of the region info file that resides just under the region directory.
static String	REGION_MERGES_DIR Temporary subdirectory of the region directory used for merges.
static String	REGION_SPLITS_DIR Temporary subdirectory of the region directory used for splits.

Method Summary

Modifier and Type	Method and Description
void	cleanupMergedRegion(RegionInfo mergedRegion) Remove merged region
org.apache.hadoop.fs.Path	commitDaughterRegion(RegionInfo regionInfo, List<org.apache.hadoop.fs.Path> allRegionFiles, MasterProcedureEnv env) Commit a daughter region, moving it from the split temporary directory to the proper location in the filesystem.
void	commitMergedRegion(List<org.apache.hadoop.fs.Path> allMergedFiles, MasterProcedureEnv env) Commit a merged region, making it ready for use.
org.apache.hadoop.fs.Path	commitStoreFile(String familyName, org.apache.hadoop.fs.Path buildPath) Move the file from a build/temp location to the main family store directory.
static HRegionFileSystem	createRegionOnFileSystem(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path tableDir, RegionInfo regionInfo) Create a new Region on file-system.
void	createSplitsDir(RegionInfo daughterA, RegionInfo daughterB) Creates region split daughter directories under the table dir.
org.apache.hadoop.fs.Path	createTempName() Generate a unique temporary Path.
org.apache.hadoop.fs.Path	createTempName(String suffix) Generate a unique temporary Path.
void	deleteFamily(String familyName) Remove the region family from disk, archiving the store files.
static void	deleteRegionFromFileSystem(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path tableDir, RegionInfo regionInfo) Remove the region from the table directory, archiving the region's hfiles.
Collection<String>	getFamilies() Returns the set of families present on disk n
org.apache.hadoop.fs.FileSystem	getFileSystem() Returns the underlying FileSystem
org.apache.hadoop.fs.Path	getRegionDir() Returns Path to the region directory.
RegionInfo	getRegionInfo() Returns the RegionInfo that describe this on-disk region view
RegionInfo	getRegionInfoForFS()
org.apache.hadoop.fs.Path	getSplitsDir(RegionInfo hri)
String	getStoragePolicyName(String familyName) Get the storage policy of the directory of CF.
org.apache.hadoop.fs.Path	getStoreDir(String familyName) Returns the directory path of the specified family
List<StoreFileInfo>	getStoreFiles(String familyName) Returns the store files available for the family.
List<StoreFileInfo>	getStoreFiles(String familyName, boolean validate) Returns the store files available for the family.
static List<org.apache.hadoop.fs.LocatedFileStatus>	getStoreFilesLocatedStatus(HRegionFileSystem regionfs, String familyName, boolean validate) Returns the store files' LocatedFileStatus which available for the family.
static org.apache.hadoop.fs.Path	getStoreHomedir(org.apache.hadoop.fs.Path tabledir, RegionInfo hri, byte[] family)
static org.apache.hadoop.fs.Path	getStoreHomedir(org.apache.hadoop.fs.Path tabledir, String encodedName, byte[] family)
org.apache.hadoop.fs.Path	getTableDir() Returns Path to the region's root directory.
org.apache.hadoop.fs.Path	getTempDir() Returns Path to the region's temp directory, used for file creations
boolean	hasReferences(String familyName) Returns true if the specified family has reference files
boolean	hasReferences(TableDescriptor htd) Check whether region has Reference file
static RegionInfo	loadRegionInfoFileContent(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path regionDir) Create a RegionInfo from the serialized version on-disk.
org.apache.hadoop.fs.Path	mergeStoreFile(RegionInfo mergingRegion, String familyName, HStoreFile f) Write out a merge reference under the given merges directory.
static HRegionFileSystem	openRegionFromFileSystem(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path tableDir, RegionInfo regionInfo, boolean readOnly) Open Region from file-system.
void	removeStoreFile(String familyName, org.apache.hadoop.fs.Path filePath) Archives the specified store file from the specified family.
void	removeStoreFiles(String familyName, Collection<HStoreFile> storeFiles) Closes and archives the specified store files from the specified family.
void	setStoragePolicy(String policyName) Set storage policy for a whole region.
void	setStoragePolicy(String familyName, String policyName) Set the directory of CF to the specified storage policy.
org.apache.hadoop.fs.Path	splitStoreFile(RegionInfo hri, String familyName, HStoreFile f, byte[] splitRow, boolean top, RegionSplitPolicy splitPolicy) Write out a split reference.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

REGION_INFO_FILE

public static final String REGION_INFO_FILE

Name of the region info file that resides just under the region directory.

See Also:

Constant Field Values

REGION_MERGES_DIR

public static final String REGION_MERGES_DIR

Temporary subdirectory of the region directory used for merges.

See Also:

Constant Field Values

REGION_SPLITS_DIR

public static final String REGION_SPLITS_DIR

Temporary subdirectory of the region directory used for splits.

See Also:

Constant Field Values

Method Detail

getFileSystem

public org.apache.hadoop.fs.FileSystem getFileSystem()

Returns the underlying FileSystem

getRegionInfo

public RegionInfo getRegionInfo()

Returns the RegionInfo that describe this on-disk region view

getRegionInfoForFS

public RegionInfo getRegionInfoForFS()

getTableDir

public org.apache.hadoop.fs.Path getTableDir()

Returns Path to the region's root directory.

getRegionDir

public org.apache.hadoop.fs.Path getRegionDir()

Returns Path to the region directory.

getTempDir

public org.apache.hadoop.fs.Path getTempDir()

Returns Path to the region's temp directory, used for file creations

getStoreDir

public org.apache.hadoop.fs.Path getStoreDir(String familyName)

Returns the directory path of the specified family

Parameters:

familyName - Column Family Name

Returns:

Path to the directory of the specified family

getStoreHomedir

public static org.apache.hadoop.fs.Path getStoreHomedir(org.apache.hadoop.fs.Path tabledir,
                                                        RegionInfo hri,
                                                        byte[] family)

Parameters:

tabledir - Path to where the table is being stored

hri - RegionInfo for the region.

family - ColumnFamilyDescriptor describing the column family

Returns:

Path to family/Store home directory.

getStoreHomedir

public static org.apache.hadoop.fs.Path getStoreHomedir(org.apache.hadoop.fs.Path tabledir,
                                                        String encodedName,
                                                        byte[] family)

Parameters:

tabledir - Path to where the table is being stored

encodedName - Encoded region name.

family - ColumnFamilyDescriptor describing the column family

Returns:

Path to family/Store home directory.

setStoragePolicy

public void setStoragePolicy(String familyName,
                             String policyName)

Set the directory of CF to the specified storage policy.
"LAZY_PERSIST", "ALL_SSD", "ONE_SSD", "HOT", "WARM", "COLD"

See HdfsConstants for more details.

Parameters:

familyName - The name of column family.

policyName - The name of the storage policy: 'HOT', 'COLD', etc. See hadoop 2.6+ org.apache.hadoop.hdfs.protocol.HdfsConstants for possible list e.g 'COLD', 'WARM', 'HOT', 'ONE_SSD', 'ALL_SSD', 'LAZY_PERSIST'.

setStoragePolicy

public void setStoragePolicy(String policyName)

Set storage policy for a whole region.
"LAZY_PERSIST", "ALL_SSD", "ONE_SSD", "HOT", "WARM", "COLD"

See HdfsConstants for more details.

Parameters:

policyName - The name of the storage policy: 'HOT', 'COLD', etc. See hadoop 2.6+ org.apache.hadoop.hdfs.protocol.HdfsConstants for possible list e.g 'COLD', 'WARM', 'HOT', 'ONE_SSD', 'ALL_SSD', 'LAZY_PERSIST'.

getStoragePolicyName

@Nullable
public String getStoragePolicyName(String familyName)

Get the storage policy of the directory of CF.

Parameters:

familyName - The name of column family.

Returns:

Storage policy name, or null if not using HFileSystem or exception thrown when trying to get policy

getStoreFiles

public List<StoreFileInfo> getStoreFiles(String familyName)
                                  throws IOException

Returns the store files available for the family. This methods performs the filtering based on the valid store files.

Parameters:

familyName - Column Family Name

Returns:

a set of StoreFileInfo for the specified family.

Throws:

IOException

getStoreFiles

public List<StoreFileInfo> getStoreFiles(String familyName,
                                         boolean validate)
                                  throws IOException

Returns the store files available for the family. This methods performs the filtering based on the valid store files.

Parameters:

familyName - Column Family Name

Returns:

a set of StoreFileInfo for the specified family.

Throws:

IOException

getStoreFilesLocatedStatus

public static List<org.apache.hadoop.fs.LocatedFileStatus> getStoreFilesLocatedStatus(HRegionFileSystem regionfs,
                                                                                      String familyName,
                                                                                      boolean validate)
                                                                               throws IOException

Returns the store files' LocatedFileStatus which available for the family. This methods performs the filtering based on the valid store files.

Parameters:

familyName - Column Family Name

Returns:

a list of store files' LocatedFileStatus for the specified family.

Throws:

IOException

hasReferences

public boolean hasReferences(String familyName)
                      throws IOException

Returns true if the specified family has reference files

Parameters:

familyName - Column Family Name

Returns:

true if family contains reference files

Throws:

IOException

hasReferences

public boolean hasReferences(TableDescriptor htd)
                      throws IOException

Check whether region has Reference file

Parameters:

htd - table desciptor of the region

Returns:

true if region has reference file

Throws:

IOException

getFamilies

public Collection<String> getFamilies()
                               throws IOException

Returns the set of families present on disk n

Throws:

IOException

deleteFamily

public void deleteFamily(String familyName)
                  throws IOException

Remove the region family from disk, archiving the store files.

Parameters:

familyName - Column Family Name

Throws:

IOException - if an error occours during the archiving

createTempName

public org.apache.hadoop.fs.Path createTempName()

Generate a unique temporary Path. Used in conjuction with commitStoreFile() to get a safer file creation. Path file = fs.createTempName(); ...StoreFile.Writer(file)... fs.commitStoreFile("family", file);

Returns:

Unique Path of the temporary file

createTempName

public org.apache.hadoop.fs.Path createTempName(String suffix)

Generate a unique temporary Path. Used in conjuction with commitStoreFile() to get a safer file creation. Path file = fs.createTempName(); ...StoreFile.Writer(file)... fs.commitStoreFile("family", file);

Parameters:

suffix - extra information to append to the generated name

Returns:

Unique Path of the temporary file

commitStoreFile

public org.apache.hadoop.fs.Path commitStoreFile(String familyName,
                                                 org.apache.hadoop.fs.Path buildPath)
                                          throws IOException

Move the file from a build/temp location to the main family store directory.

Parameters:

familyName - Family that will gain the file

buildPath - Path to the file to commit.

Returns:

The new Path of the committed file

Throws:

IOException

removeStoreFile

public void removeStoreFile(String familyName,
                            org.apache.hadoop.fs.Path filePath)
                     throws IOException

Archives the specified store file from the specified family.

Parameters:

familyName - Family that contains the store files

filePath - Path to the store file to remove

Throws:

IOException - if the archiving fails

removeStoreFiles

public void removeStoreFiles(String familyName,
                             Collection<HStoreFile> storeFiles)
                      throws IOException

Closes and archives the specified store files from the specified family.

Parameters:

familyName - Family that contains the store files

storeFiles - set of store files to remove

Throws:

IOException - if the archiving fails

getSplitsDir

public org.apache.hadoop.fs.Path getSplitsDir(RegionInfo hri)

commitDaughterRegion

public org.apache.hadoop.fs.Path commitDaughterRegion(RegionInfo regionInfo,
                                                      List<org.apache.hadoop.fs.Path> allRegionFiles,
                                                      MasterProcedureEnv env)
                                               throws IOException

Commit a daughter region, moving it from the split temporary directory to the proper location in the filesystem.

Parameters:

regionInfo - daughter RegionInfo

Throws:

IOException

createSplitsDir

public void createSplitsDir(RegionInfo daughterA,
                            RegionInfo daughterB)
                     throws IOException

Creates region split daughter directories under the table dir. If the daughter regions already exist, for example, in the case of a recovery from a previous failed split procedure, this method deletes the given region dir recursively, then recreates it again.

Throws:

IOException

splitStoreFile

public org.apache.hadoop.fs.Path splitStoreFile(RegionInfo hri,
                                                String familyName,
                                                HStoreFile f,
                                                byte[] splitRow,
                                                boolean top,
                                                RegionSplitPolicy splitPolicy)
                                         throws IOException

Write out a split reference. Package local so it doesnt leak out of regionserver.

Parameters:

hri - RegionInfo of the destination

familyName - Column Family Name

f - File to split.

splitRow - Split Row

top - True if we are referring to the top half of the hfile.

splitPolicy - A split policy instance; be careful! May not be full populated; e.g. if this method is invoked on the Master side, then the RegionSplitPolicy will NOT have a reference to a Region.

Returns:

Path to created reference.

Throws:

IOException

cleanupMergedRegion

public void cleanupMergedRegion(RegionInfo mergedRegion)
                         throws IOException

Remove merged region

Parameters:

mergedRegion - RegionInfo

Throws:

IOException

mergeStoreFile

public org.apache.hadoop.fs.Path mergeStoreFile(RegionInfo mergingRegion,
                                                String familyName,
                                                HStoreFile f)
                                         throws IOException

Write out a merge reference under the given merges directory.

Parameters:

mergingRegion - RegionInfo for one of the regions being merged.

familyName - Column Family Name

f - File to create reference.

Returns:

Path to created reference.

Throws:

IOException - if the merge write fails.

commitMergedRegion

public void commitMergedRegion(List<org.apache.hadoop.fs.Path> allMergedFiles,
                               MasterProcedureEnv env)
                        throws IOException

Commit a merged region, making it ready for use.

Throws:

IOException

loadRegionInfoFileContent

public static RegionInfo loadRegionInfoFileContent(org.apache.hadoop.fs.FileSystem fs,
                                                   org.apache.hadoop.fs.Path regionDir)
                                            throws IOException

Create a RegionInfo from the serialized version on-disk.

Parameters:

fs - FileSystem that contains the Region Info file

regionDir - Path to the Region Directory that contains the Info file

Returns:

An RegionInfo instance gotten from the Region Info file.

Throws:

IOException - if an error occurred during file open/read operation.

createRegionOnFileSystem

public static HRegionFileSystem createRegionOnFileSystem(org.apache.hadoop.conf.Configuration conf,
                                                         org.apache.hadoop.fs.FileSystem fs,
                                                         org.apache.hadoop.fs.Path tableDir,
                                                         RegionInfo regionInfo)
                                                  throws IOException

Create a new Region on file-system.

Parameters:

conf - the Configuration to use

fs - FileSystem from which to add the region

tableDir - Path to where the table is being stored

regionInfo - RegionInfo for region to be added

Throws:

IOException - if the region creation fails due to a FileSystem exception.

openRegionFromFileSystem

public static HRegionFileSystem openRegionFromFileSystem(org.apache.hadoop.conf.Configuration conf,
                                                         org.apache.hadoop.fs.FileSystem fs,
                                                         org.apache.hadoop.fs.Path tableDir,
                                                         RegionInfo regionInfo,
                                                         boolean readOnly)
                                                  throws IOException

Open Region from file-system.

Parameters:

conf - the Configuration to use

fs - FileSystem from which to add the region

tableDir - Path to where the table is being stored

regionInfo - RegionInfo for region to be added

readOnly - True if you don't want to edit the region data

Throws:

IOException - if the region creation fails due to a FileSystem exception.

deleteRegionFromFileSystem

public static void deleteRegionFromFileSystem(org.apache.hadoop.conf.Configuration conf,
                                              org.apache.hadoop.fs.FileSystem fs,
                                              org.apache.hadoop.fs.Path tableDir,
                                              RegionInfo regionInfo)
                                       throws IOException

Remove the region from the table directory, archiving the region's hfiles.

Parameters:

conf - the Configuration to use

fs - FileSystem from which to remove the region

tableDir - Path to where the table is being stored

regionInfo - RegionInfo for region to be deleted

Throws:

IOException - if the request cannot be completed