com.ksyun.kmr.hadoop.fs.ks3

类 Ks3FileSystem

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.FileSystem

com.ksyun.kmr.hadoop.fs.ks3.Ks3FileSystem

所有已实现的接口:

Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable

public class Ks3FileSystem
extends org.apache.hadoop.fs.FileSystem

Ks3FileSystem: Hadoop File System Ks3 Implementation KS3 URI Format ks3://[AccessKey]:[AccessSecret]@[Bucket]/[objectKey]

About folder structure: using object key a/b.txt to represent a file within folder There will not having fake object to represent folder if the folder is not empty; but if a folder is empty, then will have a empty object to represent the folder, i.e. "a/c/"

嵌套类概要

从类继承的嵌套类/接口 org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.FileSystem.DirectoryEntries, org.apache.hadoop.fs.FileSystem.Statistics

字段概要

字段

限定符和类型	字段和说明
long	ks3BlockSize
boolean	ks3CheckSub
URI	uri
org.apache.hadoop.fs.Path	workingDir

从类继承的字段 org.apache.hadoop.fs.FileSystem

DEFAULT_FS, FS_DEFAULT_NAME_KEY, SHUTDOWN_HOOK_PRIORITY, statistics, TRASH_PREFIX, USER_HOME_PREFIX

构造器概要

构造器

构造器和说明
Ks3FileSystem()

方法概要

限定符和类型	方法和说明
org.apache.hadoop.fs.FSDataOutputStream	append(org.apache.hadoop.fs.Path f, int bufferSize, org.apache.hadoop.util.Progressable progress)
boolean	checkNotEmptyDir(org.apache.hadoop.fs.Path path)
org.apache.hadoop.fs.FSDataOutputStream	create(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress)
void	createFakeDirectory(String objectName)
org.apache.hadoop.fs.FSDataOutputStream	createNonRecursive(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission permission, EnumSet<org.apache.hadoop.fs.CreateFlag> flags, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress)
static long	dateToLong(Date date)
boolean	delete(org.apache.hadoop.fs.Path f, boolean recursive)
boolean	delete(org.apache.hadoop.fs.Path f, boolean recursive, Ks3FileStatus status)
long	getDefaultBlockSize(org.apache.hadoop.fs.Path f)
org.apache.hadoop.fs.Path	getDirectOutputPath(org.apache.hadoop.fs.Path qualifiedPath)
Ks3FileStatus	getFileStatus(org.apache.hadoop.fs.Path f)
Ks3FileStatus	getFileStatus(org.apache.hadoop.fs.Path f, boolean checkSub)
String	getScheme()
Ks3FileSystemStore	getStore()
URI	getUri()
org.apache.hadoop.fs.Path	getWorkingDirectory()
void	initialize(URI name, org.apache.hadoop.conf.Configuration conf) Called after a new FileSystem instance is constructed.
boolean	isNotEmptyDir(Ks3FileStatus ks3FileStatus)
org.apache.hadoop.fs.Path	keyToPath(String key)
org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus>	listFiles(org.apache.hadoop.fs.Path f, boolean recursive)
protected org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus>	listLocatedStatus(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.PathFilter filter)
org.apache.hadoop.fs.FileStatus[]	listStatus(org.apache.hadoop.fs.Path f)
org.apache.hadoop.fs.FileStatus[]	listStatusWithFilter(org.apache.hadoop.fs.Path f, boolean isFlat, org.apache.hadoop.fs.PathFilter... filters)
boolean	mkdirs(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission permission)
boolean	objectRepresentsDirectory(String name, long size)
org.apache.hadoop.fs.FSDataInputStream	open(org.apache.hadoop.fs.Path f, int bufferSize)
String	pathToKey(org.apache.hadoop.fs.Path path)
static String	pathToKey(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.Path workingDir)
boolean	rename(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst) Renames Path src to Path dst Warning: KS3 does not support renames.
boolean	rename0(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst)
void	setWorkingDirectory(org.apache.hadoop.fs.Path new_dir)
org.apache.hadoop.fs.Path	toQ(org.apache.hadoop.fs.Path f)
void	validatePath(org.apache.hadoop.fs.Path f)

从类继承的方法 org.apache.hadoop.fs.FileSystem

access, addDelegationTokens, append, append, appendFile, areSymlinksEnabled, cancelDeleteOnExit, canonicalizeUri, checkPath, clearStatistics, close, closeAll, closeAllForUGI, completeLocalOutput, concat, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createFile, createNewFile, createNonRecursive, createNonRecursive, createPathHandle, createSnapshot, createSnapshot, createSymlink, delete, deleteOnExit, deleteSnapshot, enableSymlinks, exists, fixRelativePart, get, get, get, getAclStatus, getAllStatistics, getAllStoragePolicies, getBlockSize, getCanonicalServiceName, getCanonicalUri, getChildFileSystems, getContentSummary, getDefaultBlockSize, getDefaultPort, getDefaultReplication, getDefaultReplication, getDefaultUri, getDelegationToken, getFileBlockLocations, getFileBlockLocations, getFileChecksum, getFileChecksum, getFileLinkStatus, getFileSystemClass, getFSofPath, getGlobalStorageStatistics, getHomeDirectory, getInitialWorkingDirectory, getLength, getLinkTarget, getLocal, getName, getNamed, getPathHandle, getQuotaUsage, getReplication, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getStoragePolicy, getStorageStatistics, getTrashRoot, getTrashRoots, getUsed, getUsed, getXAttr, getXAttrs, getXAttrs, globStatus, globStatus, isDirectory, isFile, listCorruptFileBlocks, listLocatedStatus, listStatus, listStatus, listStatus, listStatusBatch, listStatusIterator, listXAttrs, makeQualified, mkdirs, mkdirs, modifyAclEntries, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, newInstance, newInstance, newInstance, newInstanceLocal, open, open, open, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, processDeleteOnExit, removeAcl, removeAclEntries, removeDefaultAcl, removeXAttr, rename, renameSnapshot, resolveLink, resolvePath, setAcl, setDefaultUri, setDefaultUri, setOwner, setPermission, setReplication, setStoragePolicy, setTimes, setVerifyChecksum, setWriteChecksum, setXAttr, setXAttr, startLocalOutput, supportsSymlinks, truncate, unsetStoragePolicy

从类继承的方法 org.apache.hadoop.conf.Configured

getConf, setConf

从类继承的方法 java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

字段详细资料

uri

public URI uri

workingDir

public org.apache.hadoop.fs.Path workingDir

ks3BlockSize

public long ks3BlockSize

ks3CheckSub

public boolean ks3CheckSub

构造器详细资料

Ks3FileSystem

public Ks3FileSystem()

方法详细资料

getStore

public Ks3FileSystemStore getStore()

initialize

public void initialize(URI name,
                       org.apache.hadoop.conf.Configuration conf)
                throws IOException

Called after a new FileSystem instance is constructed.

覆盖:

initialize 在类中 org.apache.hadoop.fs.FileSystem

参数:

name - a uri whose authority section names the host, port, etc. for this FileSystem

conf - the configuration

抛出:

IOException

getScheme

public String getScheme()

覆盖:

getScheme 在类中 org.apache.hadoop.fs.FileSystem

getUri

public URI getUri()

指定者:

getUri 在类中 org.apache.hadoop.fs.FileSystem

open

public org.apache.hadoop.fs.FSDataInputStream open(org.apache.hadoop.fs.Path f,
                                                   int bufferSize)
                                            throws IOException

指定者:

open 在类中 org.apache.hadoop.fs.FileSystem

抛出:

IOException

create

public org.apache.hadoop.fs.FSDataOutputStream create(org.apache.hadoop.fs.Path f,
                                                      org.apache.hadoop.fs.permission.FsPermission permission,
                                                      boolean overwrite,
                                                      int bufferSize,
                                                      short replication,
                                                      long blockSize,
                                                      org.apache.hadoop.util.Progressable progress)
                                               throws IOException

指定者:

create 在类中 org.apache.hadoop.fs.FileSystem

抛出:

IOException

createNonRecursive

public org.apache.hadoop.fs.FSDataOutputStream createNonRecursive(org.apache.hadoop.fs.Path path,
                                                                  org.apache.hadoop.fs.permission.FsPermission permission,
                                                                  EnumSet<org.apache.hadoop.fs.CreateFlag> flags,
                                                                  int bufferSize,
                                                                  short replication,
                                                                  long blockSize,
                                                                  org.apache.hadoop.util.Progressable progress)
                                                           throws IOException

覆盖:

createNonRecursive 在类中 org.apache.hadoop.fs.FileSystem

抛出:

FileNotFoundException - if the parent directory is not present -or is not a directory.

IOException

append

public org.apache.hadoop.fs.FSDataOutputStream append(org.apache.hadoop.fs.Path f,
                                                      int bufferSize,
                                                      org.apache.hadoop.util.Progressable progress)
                                               throws IOException

指定者:

append 在类中 org.apache.hadoop.fs.FileSystem

抛出:

IOException

rename

public boolean rename(org.apache.hadoop.fs.Path src,
                      org.apache.hadoop.fs.Path dst)
               throws IOException

Renames Path src to Path dst

Warning: KS3 does not support renames. This method does a copy which can take KS3 some time to execute with large files and directories. Since there is no Progressable passed in, this can time out jobs. Cases srcType dstType Result 1 file/dir N/A Copy 2 file/dir file Fail 3 file dir Fail 4 dir dir(not empty) Fail (whatever src type, is dst is not an empty dir, then will fail) 5 dir dir(empty) copy

指定者:

rename 在类中 org.apache.hadoop.fs.FileSystem

参数:

src - path to be renamed

dst - new path after rename

返回:

true if rename is successful

抛出:

IOException - on failure

rename0

public boolean rename0(org.apache.hadoop.fs.Path src,
                       org.apache.hadoop.fs.Path dst)
                throws IOException

抛出:

IOException

delete

public boolean delete(org.apache.hadoop.fs.Path f,
                      boolean recursive)
               throws IOException

指定者:

delete 在类中 org.apache.hadoop.fs.FileSystem

抛出:

IOException

delete

public boolean delete(org.apache.hadoop.fs.Path f,
                      boolean recursive,
                      Ks3FileStatus status)
               throws IOException

抛出:

IOException

listStatus

public org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.fs.Path f)
                                             throws IOException

指定者:

listStatus 在类中 org.apache.hadoop.fs.FileSystem

抛出:

IOException

listStatusWithFilter

public org.apache.hadoop.fs.FileStatus[] listStatusWithFilter(org.apache.hadoop.fs.Path f,
                                                              boolean isFlat,
                                                              org.apache.hadoop.fs.PathFilter... filters)
                                                       throws IOException

抛出:

IOException

listFiles

public org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> listFiles(org.apache.hadoop.fs.Path f,
                                                                                             boolean recursive)
                                                                                      throws IOException

覆盖:

listFiles 在类中 org.apache.hadoop.fs.FileSystem

抛出:

IOException

listLocatedStatus

protected org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> listLocatedStatus(org.apache.hadoop.fs.Path f,
                                                                                                        org.apache.hadoop.fs.PathFilter filter)
                                                                                                 throws IOException

覆盖:

listLocatedStatus 在类中 org.apache.hadoop.fs.FileSystem

抛出:

IOException

setWorkingDirectory

public void setWorkingDirectory(org.apache.hadoop.fs.Path new_dir)

指定者:

setWorkingDirectory 在类中 org.apache.hadoop.fs.FileSystem

getWorkingDirectory

public org.apache.hadoop.fs.Path getWorkingDirectory()

指定者:

getWorkingDirectory 在类中 org.apache.hadoop.fs.FileSystem

mkdirs

public boolean mkdirs(org.apache.hadoop.fs.Path f,
                      org.apache.hadoop.fs.permission.FsPermission permission)
               throws IOException

指定者:

mkdirs 在类中 org.apache.hadoop.fs.FileSystem

抛出:

IOException

validatePath

public void validatePath(org.apache.hadoop.fs.Path f)
                  throws IOException

抛出:

IOException

getFileStatus

public Ks3FileStatus getFileStatus(org.apache.hadoop.fs.Path f)
                            throws IOException

指定者:

getFileStatus 在类中 org.apache.hadoop.fs.FileSystem

抛出:

IOException

getFileStatus

public Ks3FileStatus getFileStatus(org.apache.hadoop.fs.Path f,
                                   boolean checkSub)
                            throws IOException

抛出:

IOException

getDefaultBlockSize

public long getDefaultBlockSize(org.apache.hadoop.fs.Path f)

覆盖:

getDefaultBlockSize 在类中 org.apache.hadoop.fs.FileSystem

objectRepresentsDirectory

public boolean objectRepresentsDirectory(String name,
                                         long size)

dateToLong

public static long dateToLong(Date date)

createFakeDirectory

public void createFakeDirectory(String objectName)

pathToKey

public String pathToKey(org.apache.hadoop.fs.Path path)

pathToKey

public static String pathToKey(org.apache.hadoop.fs.Path path,
                               org.apache.hadoop.fs.Path workingDir)

keyToPath

public org.apache.hadoop.fs.Path keyToPath(String key)

getDirectOutputPath

public org.apache.hadoop.fs.Path getDirectOutputPath(org.apache.hadoop.fs.Path qualifiedPath)

toQ

public org.apache.hadoop.fs.Path toQ(org.apache.hadoop.fs.Path f)

isNotEmptyDir

public boolean isNotEmptyDir(Ks3FileStatus ks3FileStatus)
                      throws IOException

抛出:

IOException

checkNotEmptyDir

public boolean checkNotEmptyDir(org.apache.hadoop.fs.Path path)