org.apache.hadoop.hbase.fs

Class HFileSystem

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.FilterFileSystem

org.apache.hadoop.hbase.fs.HFileSystem

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable, org.apache.hadoop.fs.PathCapabilities, org.apache.hadoop.security.token.DelegationTokenIssuer

@InterfaceAudience.Private
public class HFileSystem
extends org.apache.hadoop.fs.FilterFileSystem

An encapsulation for the FileSystem object that hbase uses to access data. This class allows the flexibility of using separate filesystem objects for reading and writing hfiles and wals.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.FileSystem.DirectoryEntries, org.apache.hadoop.fs.FileSystem.DirListingIterator<T extends org.apache.hadoop.fs.FileStatus>, org.apache.hadoop.fs.FileSystem.Statistics

Field Summary

Fields

Modifier and Type	Field and Description
static org.slf4j.Logger	LOG

Fields inherited from class org.apache.hadoop.fs.FilterFileSystem

fs, swapScheme

Fields inherited from class org.apache.hadoop.fs.FileSystem

DEFAULT_FS, FS_DEFAULT_NAME_KEY, SHUTDOWN_HOOK_PRIORITY, statistics, TRASH_PREFIX, USER_HOME_PREFIX

Constructor Summary

Constructors

Constructor and Description
HFileSystem(org.apache.hadoop.conf.Configuration conf, boolean useHBaseChecksum) Create a FileSystem object for HBase regionservers.
HFileSystem(org.apache.hadoop.fs.FileSystem fs) Wrap a FileSystem object within a HFileSystem.

Method Summary

Modifier and Type	Method and Description
static boolean	addLocationsOrderInterceptor(org.apache.hadoop.conf.Configuration conf)
void	close() Close this filesystem object
org.apache.hadoop.fs.FSDataOutputStream	createNonRecursive(org.apache.hadoop.fs.Path f, boolean overwrite, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress) The org.apache.hadoop.fs.FilterFileSystem does not yet support createNonRecursive.
static org.apache.hadoop.fs.FileSystem	get(org.apache.hadoop.conf.Configuration conf) Create a new HFileSystem object, similar to FileSystem.get().
org.apache.hadoop.fs.FileSystem	getBackingFs() Returns the underlying filesystem
org.apache.hadoop.fs.FileSystem	getNoChecksumFs() Returns the filesystem that is specially setup for doing reads from storage.
String	getStoragePolicyName(org.apache.hadoop.fs.Path path) Get the storage policy of the source path (directory/file).
void	setStoragePolicy(org.apache.hadoop.fs.Path path, String policyName) Set the source path (directory/file) to the specified storage policy.
boolean	useHBaseChecksum() Are we verifying checksums in HBase?

Methods inherited from class org.apache.hadoop.fs.FilterFileSystem

access, append, appendFile, canonicalizeUri, checkPath, completeLocalOutput, concat, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, create, create, createFile, createNonRecursive, createPathHandle, createSnapshot, createSymlink, delete, deleteSnapshot, getAclStatus, getAllStoragePolicies, getCanonicalUri, getChildFileSystems, getConf, getDefaultBlockSize, getDefaultBlockSize, getDefaultReplication, getDefaultReplication, getFileBlockLocations, getFileChecksum, getFileChecksum, getFileLinkStatus, getFileStatus, getHomeDirectory, getInitialWorkingDirectory, getLinkTarget, getRawFileSystem, getServerDefaults, getServerDefaults, getStatus, getStoragePolicy, getTrashRoot, getTrashRoots, getUri, getUsed, getUsed, getWorkingDirectory, getXAttr, getXAttrs, getXAttrs, hasPathCapability, initialize, listCorruptFileBlocks, listLocatedStatus, listLocatedStatus, listStatus, listStatusIterator, listXAttrs, makeQualified, mkdirs, mkdirs, modifyAclEntries, msync, open, open, primitiveCreate, primitiveMkdir, removeAcl, removeAclEntries, removeDefaultAcl, removeXAttr, rename, rename, renameSnapshot, resolveLink, resolvePath, satisfyStoragePolicy, setAcl, setOwner, setPermission, setReplication, setTimes, setVerifyChecksum, setWorkingDirectory, setWriteChecksum, setXAttr, setXAttr, startLocalOutput, supportsSymlinks, truncate, unsetStoragePolicy

Methods inherited from class org.apache.hadoop.fs.FileSystem

append, append, areSymlinksEnabled, cancelDeleteOnExit, clearStatistics, closeAll, closeAllForUGI, copyFromLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, createNewFile, createNonRecursive, createSnapshot, delete, deleteOnExit, enableSymlinks, exists, fixRelativePart, get, get, getAdditionalTokenIssuers, getAllStatistics, getBlockSize, getCanonicalServiceName, getContentSummary, getDefaultPort, getDefaultUri, getDelegationToken, getFileBlockLocations, getFileSystemClass, getFSofPath, getGlobalStorageStatistics, getLength, getLocal, getName, getNamed, getPathHandle, getQuotaUsage, getReplication, getScheme, getStatistics, getStatistics, getStatus, getStorageStatistics, globStatus, globStatus, isDirectory, isFile, listFiles, listStatus, listStatus, listStatus, listStatusBatch, mkdirs, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, newInstance, newInstance, newInstance, newInstanceLocal, open, open, primitiveMkdir, printStatistics, processDeleteOnExit, setDefaultUri, setDefaultUri, setQuota, setQuotaByStorageType

Methods inherited from class org.apache.hadoop.conf.Configured

setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.security.token.DelegationTokenIssuer

addDelegationTokens, collectDelegationTokens

Field Detail

LOG

public static final org.slf4j.Logger LOG

Constructor Detail

HFileSystem

public HFileSystem(org.apache.hadoop.conf.Configuration conf,
                   boolean useHBaseChecksum)
            throws IOException

Create a FileSystem object for HBase regionservers.

Parameters:

conf - The configuration to be used for the filesystem

useHBaseChecksum - if true, then use checksum verfication in hbase, otherwise delegate checksum verification to the FileSystem.

Throws:

IOException

HFileSystem

public HFileSystem(org.apache.hadoop.fs.FileSystem fs)

Wrap a FileSystem object within a HFileSystem. The noChecksumFs and writefs are both set to be the same specified fs. Do not verify hbase-checksums while reading data from filesystem.

Parameters:

fs - Set the noChecksumFs and writeFs to this specified filesystem.

Method Detail

getNoChecksumFs

public org.apache.hadoop.fs.FileSystem getNoChecksumFs()

Returns the filesystem that is specially setup for doing reads from storage. This object avoids doing checksum verifications for reads.

Returns:

The FileSystem object that can be used to read data from files.

getBackingFs

public org.apache.hadoop.fs.FileSystem getBackingFs()
                                             throws IOException

Returns the underlying filesystem

Returns:

The underlying FileSystem for this FilterFileSystem object.

Throws:

IOException

setStoragePolicy

public void setStoragePolicy(org.apache.hadoop.fs.Path path,
                             String policyName)

Set the source path (directory/file) to the specified storage policy.

Overrides:

setStoragePolicy in class org.apache.hadoop.fs.FilterFileSystem

Parameters:

path - The source path (directory/file).

policyName - The name of the storage policy: 'HOT', 'COLD', etc. See see hadoop 2.6+ org.apache.hadoop.hdfs.protocol.HdfsConstants for possible list e.g 'COLD', 'WARM', 'HOT', 'ONE_SSD', 'ALL_SSD', 'LAZY_PERSIST'.

getStoragePolicyName

@Nullable
public String getStoragePolicyName(org.apache.hadoop.fs.Path path)

Get the storage policy of the source path (directory/file).

Parameters:

path - The source path (directory/file).

Returns:

Storage policy name, or null if not using DistributedFileSystem or exception thrown when trying to get policy

useHBaseChecksum

public boolean useHBaseChecksum()

Are we verifying checksums in HBase?

Returns:

True, if hbase is configured to verify checksums, otherwise false.

close

public void close()
           throws IOException

Close this filesystem object

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class org.apache.hadoop.fs.FilterFileSystem

Throws:

IOException

addLocationsOrderInterceptor

public static boolean addLocationsOrderInterceptor(org.apache.hadoop.conf.Configuration conf)
                                            throws IOException

Throws:

IOException

get

public static org.apache.hadoop.fs.FileSystem get(org.apache.hadoop.conf.Configuration conf)
                                           throws IOException

Create a new HFileSystem object, similar to FileSystem.get(). This returns a filesystem object that avoids checksum verification in the filesystem for hfileblock-reads. For these blocks, checksum verification is done by HBase.

Throws:

IOException

createNonRecursive

public org.apache.hadoop.fs.FSDataOutputStream createNonRecursive(org.apache.hadoop.fs.Path f,
                                                                  boolean overwrite,
                                                                  int bufferSize,
                                                                  short replication,
                                                                  long blockSize,
                                                                  org.apache.hadoop.util.Progressable progress)
                                                           throws IOException

The org.apache.hadoop.fs.FilterFileSystem does not yet support createNonRecursive. This is a hadoop bug and when it is fixed in Hadoop, this definition will go away.

Overrides:

createNonRecursive in class org.apache.hadoop.fs.FileSystem

Throws:

IOException