org.apache.hadoop.fs.obs

Class OBSFileSystem

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.obs.OBSFileSystem

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable, org.apache.hadoop.fs.PathCapabilities, org.apache.hadoop.security.token.DelegationTokenIssuer

@InterfaceAudience.Private
 @InterfaceStability.Evolving
public final class OBSFileSystem
extends org.apache.hadoop.fs.FileSystem

The core OBS Filesystem implementation.

This subclass is marked as private as code should not be creating it directly; use FileSystem.get(Configuration) and variants to create one.

If cast to OBSFileSystem, extra methods and features may be accessed. Consider those private and unstable.

Because it prints some of the state of the instrumentation, the output of toString() must also be considered unstable.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.FileSystem.DirectoryEntries, org.apache.hadoop.fs.FileSystem.DirListingIterator<T extends org.apache.hadoop.fs.FileStatus>, org.apache.hadoop.fs.FileSystem.Statistics

Field Summary

Fields

Modifier and Type	Field and Description
static org.slf4j.Logger	LOG Class logger.

Fields inherited from class org.apache.hadoop.fs.FileSystem

DEFAULT_FS, FS_DEFAULT_NAME_KEY, SHUTDOWN_HOOK_PRIORITY, statistics, TRASH_PREFIX, USER_HOME_PREFIX

Fields inherited from interface org.apache.hadoop.security.token.DelegationTokenIssuer

TOKEN_LOG

Constructor Summary

Constructors

Constructor and Description
OBSFileSystem()

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.fs.FSDataOutputStream	append(org.apache.hadoop.fs.Path f, int bufferSize, org.apache.hadoop.util.Progressable progress) Append to an existing file (optional operation).
protected URI	canonicalizeUri(URI rawUri) Canonicalize the given URI.
void	checkPath(org.apache.hadoop.fs.Path path) Check that a Path belongs to this FileSystem.
void	close() Close the filesystem.
void	copyFromLocalFile(boolean delSrc, boolean overwrite, org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst) Copy the src file on the local disk to the filesystem at the given dst name.
org.apache.hadoop.fs.FSDataOutputStream	create(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission permission, boolean overwrite, int bufferSize, short replication, long blkSize, org.apache.hadoop.util.Progressable progress) Create an FSDataOutputStream at the indicated Path with write-progress reporting.
org.apache.hadoop.fs.FSDataOutputStream	create(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission permission, EnumSet<org.apache.hadoop.fs.CreateFlag> flags, int bufferSize, short replication, long blkSize, org.apache.hadoop.util.Progressable progress, org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt) Create an FSDataOutputStream at the indicated Path with write-progress reporting.
org.apache.hadoop.fs.FSDataOutputStream	createNonRecursive(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission permission, EnumSet<org.apache.hadoop.fs.CreateFlag> flags, int bufferSize, short replication, long blkSize, org.apache.hadoop.util.Progressable progress) Open an FSDataOutputStream at the indicated Path with write-progress reporting.
boolean	delete(org.apache.hadoop.fs.Path f, boolean recursive) Delete a Path.
boolean	exists(org.apache.hadoop.fs.Path f) Check if a path exists.
String	getCanonicalServiceName() Override getCanonicalServiceName and return null since delegation token is not supported.
org.apache.hadoop.fs.ContentSummary	getContentSummary(org.apache.hadoop.fs.Path f) Return the ContentSummary of a given Path.
long	getDefaultBlockSize() Deprecated. use getDefaultBlockSize(Path) instead
long	getDefaultBlockSize(org.apache.hadoop.fs.Path f) Imitate HDFS to return the number of bytes that large input files should be optimally split into to minimize I/O time.
int	getDefaultPort() Return the default port for this FileSystem.
org.apache.hadoop.fs.FileStatus	getFileStatus(org.apache.hadoop.fs.Path f) Return a file status object that represents the path.
String	getScheme() Return the protocol scheme for the FileSystem.
URI	getUri() Return a URI whose scheme and authority identify this FileSystem.
org.apache.hadoop.fs.Path	getWorkingDirectory() Return the current working directory for the given file system.
void	initialize(URI name, org.apache.hadoop.conf.Configuration originalConf) Initialize a FileSystem.
org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus>	listFiles(org.apache.hadoop.fs.Path f, boolean recursive) List the statuses and block locations of the files in the given path.
org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus>	listLocatedStatus(org.apache.hadoop.fs.Path f) List the statuses of the files/directories in the given path if the path is a directory.
org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus>	listLocatedStatus(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.PathFilter filter) List a directory.
org.apache.hadoop.fs.FileStatus[]	listStatus(org.apache.hadoop.fs.Path f) List the statuses of the files/directories in the given path if the path is a directory.
org.apache.hadoop.fs.FileStatus[]	listStatus(org.apache.hadoop.fs.Path f, boolean recursive) This public interface is provided specially for Huawei MRS.
boolean	mkdirs(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission permission) Make the given path and all non-existent parents into directories.
org.apache.hadoop.fs.FSDataInputStream	open(org.apache.hadoop.fs.Path f, int bufferSize) Open an FSDataInputStream at the indicated Path.
boolean	rename(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst) Rename Path src to Path dst.
void	setWorkingDirectory(org.apache.hadoop.fs.Path newDir) Set the current working directory for the file system.
String	toString() Return a string that describes this filesystem instance.

Methods inherited from class org.apache.hadoop.fs.FileSystem

access, append, append, append, append, appendFile, areSymlinksEnabled, cancelDeleteOnExit, clearStatistics, closeAll, closeAllForUGI, completeLocalOutput, concat, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, createDataInputStreamBuilder, createDataInputStreamBuilder, createDataOutputStreamBuilder, createFile, createMultipartUploader, createNewFile, createNonRecursive, createNonRecursive, createPathHandle, createSnapshot, createSnapshot, createSymlink, delete, deleteOnExit, deleteSnapshot, enableSymlinks, fixRelativePart, get, get, get, getAclStatus, getAdditionalTokenIssuers, getAllStatistics, getAllStoragePolicies, getBlockSize, getCanonicalUri, getChildFileSystems, getDefaultReplication, getDefaultReplication, getDefaultUri, getDelegationToken, getEnclosingRoot, getFileBlockLocations, getFileBlockLocations, getFileChecksum, getFileChecksum, getFileLinkStatus, getFileSystemClass, getFSofPath, getGlobalStorageStatistics, getHomeDirectory, getInitialWorkingDirectory, getLength, getLinkTarget, getLocal, getName, getNamed, getPathHandle, getQuotaUsage, getReplication, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getStoragePolicy, getStorageStatistics, getTrashRoot, getTrashRoots, getUsed, getUsed, getXAttr, getXAttrs, getXAttrs, globStatus, globStatus, hasPathCapability, isDirectory, isFile, listCorruptFileBlocks, listStatus, listStatus, listStatus, listStatusBatch, listStatusIterator, listXAttrs, makeQualified, mkdirs, mkdirs, modifyAclEntries, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, msync, newInstance, newInstance, newInstance, newInstanceLocal, open, open, open, openFile, openFile, openFileWithOptions, openFileWithOptions, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, processDeleteOnExit, removeAcl, removeAclEntries, removeDefaultAcl, removeXAttr, rename, renameSnapshot, resolveLink, resolvePath, satisfyStoragePolicy, setAcl, setDefaultUri, setDefaultUri, setOwner, setPermission, setQuota, setQuotaByStorageType, setReplication, setStoragePolicy, setTimes, setVerifyChecksum, setWriteChecksum, setXAttr, setXAttr, startLocalOutput, supportsSymlinks, truncate, unsetStoragePolicy

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.security.token.DelegationTokenIssuer

addDelegationTokens, collectDelegationTokens

Field Detail

LOG

public static final org.slf4j.Logger LOG

Class logger.

Constructor Detail

OBSFileSystem

public OBSFileSystem()

Method Detail

initialize

public void initialize(URI name,
                       org.apache.hadoop.conf.Configuration originalConf)
                throws IOException

Initialize a FileSystem. Called after a new FileSystem instance is constructed.

Overrides:

initialize in class org.apache.hadoop.fs.FileSystem

Parameters:

name - a URI whose authority section names the host, port, etc. for this FileSystem

originalConf - the configuration to use for the FS. The bucket-specific options are patched over the base ones before any use is made of the config.

Throws:

IOException

getScheme

public String getScheme()

Return the protocol scheme for the FileSystem.

Overrides:

getScheme in class org.apache.hadoop.fs.FileSystem

Returns:

"obs"

getUri

public URI getUri()

Return a URI whose scheme and authority identify this FileSystem.

Specified by:

getUri in class org.apache.hadoop.fs.FileSystem

Returns:

the URI of this filesystem.

getDefaultPort

public int getDefaultPort()

Return the default port for this FileSystem.

Overrides:

getDefaultPort in class org.apache.hadoop.fs.FileSystem

Returns:

-1 to indicate the port is undefined, which agrees with the contract of URI.getPort()

checkPath

public void checkPath(org.apache.hadoop.fs.Path path)

Check that a Path belongs to this FileSystem. Unlike the superclass, this version does not look at authority, but only hostname.

Overrides:

checkPath in class org.apache.hadoop.fs.FileSystem

Parameters:

path - the path to check

Throws:

IllegalArgumentException - if there is an FS mismatch

canonicalizeUri

protected URI canonicalizeUri(URI rawUri)

Canonicalize the given URI.

Overrides:

canonicalizeUri in class org.apache.hadoop.fs.FileSystem

Parameters:

rawUri - the URI to be canonicalized

Returns:

the canonicalized URI

open

public org.apache.hadoop.fs.FSDataInputStream open(org.apache.hadoop.fs.Path f,
                                                   int bufferSize)
                                            throws IOException

Open an FSDataInputStream at the indicated Path.

Specified by:

open in class org.apache.hadoop.fs.FileSystem

Parameters:

f - the file path to open

bufferSize - the size of the buffer to be used

Returns:

the FSDataInputStream for the file

Throws:

IOException - on any failure to open the file

create

public org.apache.hadoop.fs.FSDataOutputStream create(org.apache.hadoop.fs.Path f,
                                                      org.apache.hadoop.fs.permission.FsPermission permission,
                                                      boolean overwrite,
                                                      int bufferSize,
                                                      short replication,
                                                      long blkSize,
                                                      org.apache.hadoop.util.Progressable progress)
                                               throws IOException

Create an FSDataOutputStream at the indicated Path with write-progress reporting.

Specified by:

create in class org.apache.hadoop.fs.FileSystem

Parameters:

f - the file path to create

permission - the permission to set

overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown

bufferSize - the size of the buffer to be used

replication - required block replication for the file

blkSize - the requested block size

progress - the progress reporter

Throws:

IOException - on any failure to create the file

See Also:

FileSystem.setPermission(Path, FsPermission)

create

public org.apache.hadoop.fs.FSDataOutputStream create(org.apache.hadoop.fs.Path f,
                                                      org.apache.hadoop.fs.permission.FsPermission permission,
                                                      EnumSet<org.apache.hadoop.fs.CreateFlag> flags,
                                                      int bufferSize,
                                                      short replication,
                                                      long blkSize,
                                                      org.apache.hadoop.util.Progressable progress,
                                                      org.apache.hadoop.fs.Options.ChecksumOpt checksumOpt)
                                               throws IOException

Create an FSDataOutputStream at the indicated Path with write-progress reporting.

Overrides:

create in class org.apache.hadoop.fs.FileSystem

Parameters:

f - the file name to create

permission - permission of

flags - CreateFlags to use for this stream

bufferSize - the size of the buffer to be used

replication - required block replication for the file

blkSize - block size

progress - progress

checksumOpt - check sum option

Throws:

IOException - io exception

createNonRecursive

public org.apache.hadoop.fs.FSDataOutputStream createNonRecursive(org.apache.hadoop.fs.Path path,
                                                                  org.apache.hadoop.fs.permission.FsPermission permission,
                                                                  EnumSet<org.apache.hadoop.fs.CreateFlag> flags,
                                                                  int bufferSize,
                                                                  short replication,
                                                                  long blkSize,
                                                                  org.apache.hadoop.util.Progressable progress)
                                                           throws IOException

Open an FSDataOutputStream at the indicated Path with write-progress reporting. Same as create(), except fails if parent directory doesn't already exist.

Overrides:

createNonRecursive in class org.apache.hadoop.fs.FileSystem

Parameters:

path - the file path to create

permission - file permission

flags - CreateFlags to use for this stream

bufferSize - the size of the buffer to be used

replication - required block replication for the file

blkSize - block size

progress - the progress reporter

Throws:

IOException - IO failure

append

public org.apache.hadoop.fs.FSDataOutputStream append(org.apache.hadoop.fs.Path f,
                                                      int bufferSize,
                                                      org.apache.hadoop.util.Progressable progress)
                                               throws IOException

Append to an existing file (optional operation).

Specified by:

append in class org.apache.hadoop.fs.FileSystem

Parameters:

f - the existing file to be appended

bufferSize - the size of the buffer to be used

progress - for reporting progress if it is not null

Throws:

IOException - indicating that append is not supported

exists

public boolean exists(org.apache.hadoop.fs.Path f)
               throws IOException

Check if a path exists.

Overrides:

exists in class org.apache.hadoop.fs.FileSystem

Parameters:

f - source path

Returns:

true if the path exists

Throws:

IOException - IO failure

rename

public boolean rename(org.apache.hadoop.fs.Path src,
                      org.apache.hadoop.fs.Path dst)
               throws IOException

Rename Path src to Path dst.

Specified by:

rename in class org.apache.hadoop.fs.FileSystem

Parameters:

src - path to be renamed

dst - new path after rename

Returns:

true if rename is successful

Throws:

IOException - on IO failure

delete

public boolean delete(org.apache.hadoop.fs.Path f,
                      boolean recursive)
               throws IOException

Delete a Path. This operation is at least O(files), with added overheads to enumerate the path. It is also not atomic.

Specified by:

delete in class org.apache.hadoop.fs.FileSystem

Parameters:

f - the path to delete

recursive - if path is a directory and set to true, the directory is deleted else throws an exception. In case of a file the recursive can be set to either true or false

Returns:

true if delete is successful else false

Throws:

IOException - due to inability to delete a directory or file

listStatus

public org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.fs.Path f)
                                             throws FileNotFoundException,
                                                    IOException

List the statuses of the files/directories in the given path if the path is a directory.

Specified by:

listStatus in class org.apache.hadoop.fs.FileSystem

Parameters:

f - given path

Returns:

the statuses of the files/directories in the given patch

Throws:

FileNotFoundException - when the path does not exist

IOException - see specific implementation

listStatus

public org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.fs.Path f,
                                                    boolean recursive)
                                             throws FileNotFoundException,
                                                    IOException

This public interface is provided specially for Huawei MRS. List the statuses of the files/directories in the given path if the path is a directory. When recursive is true, iterator all objects in the given path and its sub directories.

Parameters:

f - given path

recursive - whether to iterator objects in sub direcotries

Returns:

the statuses of the files/directories in the given patch

Throws:

FileNotFoundException - when the path does not exist

IOException - see specific implementation

getWorkingDirectory

public org.apache.hadoop.fs.Path getWorkingDirectory()

Return the current working directory for the given file system.

Specified by:

getWorkingDirectory in class org.apache.hadoop.fs.FileSystem

Returns:

the directory pathname

setWorkingDirectory

public void setWorkingDirectory(org.apache.hadoop.fs.Path newDir)

Set the current working directory for the file system. All relative paths will be resolved relative to it.

Specified by:

setWorkingDirectory in class org.apache.hadoop.fs.FileSystem

Parameters:

newDir - the new working directory

mkdirs

public boolean mkdirs(org.apache.hadoop.fs.Path path,
                      org.apache.hadoop.fs.permission.FsPermission permission)
               throws IOException,
                      org.apache.hadoop.fs.FileAlreadyExistsException

Make the given path and all non-existent parents into directories. Has the semantics of Unix 'mkdir -p'. Existence of the directory hierarchy is not an error.

Specified by:

mkdirs in class org.apache.hadoop.fs.FileSystem

Parameters:

path - path to create

permission - to apply to f

Returns:

true if a directory was created

Throws:

org.apache.hadoop.fs.FileAlreadyExistsException - there is a file at the path specified

IOException - other IO problems

getFileStatus

public org.apache.hadoop.fs.FileStatus getFileStatus(org.apache.hadoop.fs.Path f)
                                              throws FileNotFoundException,
                                                     IOException

Return a file status object that represents the path.

Specified by:

getFileStatus in class org.apache.hadoop.fs.FileSystem

Parameters:

f - the path we want information from

Returns:

a FileStatus object

Throws:

FileNotFoundException - when the path does not exist

IOException - on other problems

getContentSummary

public org.apache.hadoop.fs.ContentSummary getContentSummary(org.apache.hadoop.fs.Path f)
                                                      throws FileNotFoundException,
                                                             IOException

Return the ContentSummary of a given Path.

Overrides:

getContentSummary in class org.apache.hadoop.fs.FileSystem

Parameters:

f - path to use

Returns:

the ContentSummary

Throws:

FileNotFoundException - if the path does not resolve

IOException - IO failure

copyFromLocalFile

public void copyFromLocalFile(boolean delSrc,
                              boolean overwrite,
                              org.apache.hadoop.fs.Path src,
                              org.apache.hadoop.fs.Path dst)
                       throws org.apache.hadoop.fs.FileAlreadyExistsException,
                              IOException

Copy the src file on the local disk to the filesystem at the given dst name.

Overrides:

copyFromLocalFile in class org.apache.hadoop.fs.FileSystem

Parameters:

delSrc - whether to delete the src

overwrite - whether to overwrite an existing file

src - path

dst - path

Throws:

org.apache.hadoop.fs.FileAlreadyExistsException - if the destination file exists and overwrite == false

IOException - IO problem

close

public void close()
           throws IOException

Close the filesystem. This shuts down all transfers.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class org.apache.hadoop.fs.FileSystem

Throws:

IOException - IO problem

getCanonicalServiceName

public String getCanonicalServiceName()

Override getCanonicalServiceName and return null since delegation token is not supported.

Specified by:

getCanonicalServiceName in interface org.apache.hadoop.security.token.DelegationTokenIssuer

Overrides:

getCanonicalServiceName in class org.apache.hadoop.fs.FileSystem

getDefaultBlockSize

public long getDefaultBlockSize()

Deprecated. use getDefaultBlockSize(Path) instead

Imitate HDFS to return the number of bytes that large input files should be optimally split into to minimize I/O time for compatibility.

Overrides:

getDefaultBlockSize in class org.apache.hadoop.fs.FileSystem

getDefaultBlockSize

public long getDefaultBlockSize(org.apache.hadoop.fs.Path f)

Imitate HDFS to return the number of bytes that large input files should be optimally split into to minimize I/O time. The given path will be used to locate the actual filesystem. The full path does not have to exist.

Overrides:

getDefaultBlockSize in class org.apache.hadoop.fs.FileSystem

Parameters:

f - path of file

Returns:

the default block size for the path's filesystem

toString

public String toString()

Return a string that describes this filesystem instance.

Overrides:

toString in class Object

Returns:

the string

listFiles

public org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> listFiles(org.apache.hadoop.fs.Path f,
                                                                                             boolean recursive)
                                                                                      throws FileNotFoundException,
                                                                                             IOException

List the statuses and block locations of the files in the given path. Does not guarantee to return the iterator that traverses statuses of the files in a sorted order.

 If the path is a directory,
   if recursive is false, returns files in the directory;
   if recursive is true, return files in the subtree rooted at the path.
 If the path is a file, return the file's status and block locations.
 

Overrides:

listFiles in class org.apache.hadoop.fs.FileSystem

Parameters:

f - a path

recursive - if the subdirectories need to be traversed recursively

Returns:

an iterator that traverses statuses of the files/directories in the given path

Throws:

FileNotFoundException - if path does not exist

IOException - if any I/O error occurred

listLocatedStatus

public org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> listLocatedStatus(org.apache.hadoop.fs.Path f)
                                                                                              throws FileNotFoundException,
                                                                                                     IOException

List the statuses of the files/directories in the given path if the path is a directory. Return the file's status and block locations If the path is a file.

If a returned status is a file, it contains the file's block locations.

Overrides:

listLocatedStatus in class org.apache.hadoop.fs.FileSystem

Parameters:

f - is the path

Returns:

an iterator that traverses statuses of the files/directories in the given path

Throws:

FileNotFoundException - If f does not exist

IOException - If an I/O error occurred

listLocatedStatus

public org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> listLocatedStatus(org.apache.hadoop.fs.Path f,
                                                                                                     org.apache.hadoop.fs.PathFilter filter)
                                                                                              throws FileNotFoundException,
                                                                                                     IOException

List a directory. The returned results include its block location if it is a file The results are filtered by the given path filter

Overrides:

listLocatedStatus in class org.apache.hadoop.fs.FileSystem

Parameters:

f - a path

filter - a path filter

Returns:

an iterator that traverses statuses of the files/directories in the given path

Throws:

FileNotFoundException - if f does not exist

IOException - if any I/O error occurred