org.apache.hadoop.hive.ql.io

Class SingleFileSystem

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.FileSystem

org.apache.hadoop.hive.ql.io.SingleFileSystem

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable, org.apache.hadoop.fs.PathCapabilities, org.apache.hadoop.security.token.DelegationTokenIssuer

Direct Known Subclasses:

SingleFileSystem.ABFS, SingleFileSystem.ABFSS, SingleFileSystem.ADL, SingleFileSystem.FILE, SingleFileSystem.GS, SingleFileSystem.HDFS, SingleFileSystem.O3FS, SingleFileSystem.PFILE, SingleFileSystem.S3A

public abstract class SingleFileSystem
extends org.apache.hadoop.fs.FileSystem

Implements an abstraction layer to show files in a single directory. Suppose the filesystem has a directory in which there are multiple files: file://somedir/f1.txt file://somedir/f2.txt In case of Hive the content of a directory may be inside a table. To give a way to show a single file as a single file in a directory it could be specified: sfs+file://somedir/f1.txt/#SINGLEFILE# This will be a directory containing only the f1.txt and nothing else.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SingleFileSystem.ABFS
static class	SingleFileSystem.ABFSS
static class	SingleFileSystem.ADL
static class	SingleFileSystem.FILE
static class	SingleFileSystem.GS
static class	SingleFileSystem.HDFS
static class	SingleFileSystem.O3FS
static class	SingleFileSystem.PFILE
static class	SingleFileSystem.S3A

Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.FileSystem.DirectoryEntries, org.apache.hadoop.fs.FileSystem.DirListingIterator<T extends org.apache.hadoop.fs.FileStatus>, org.apache.hadoop.fs.FileSystem.Statistics

Field Summary

Fields inherited from class org.apache.hadoop.fs.FileSystem

DEFAULT_FS, FS_DEFAULT_NAME_KEY, LOG, SHUTDOWN_HOOK_PRIORITY, statistics, TRASH_PREFIX, USER_HOME_PREFIX

Constructor Summary

Constructors

Constructor and Description
SingleFileSystem()

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.fs.FSDataOutputStream	append(org.apache.hadoop.fs.Path upperPath, int bufferSize, org.apache.hadoop.util.Progressable progress)
org.apache.hadoop.fs.FSDataOutputStream	create(org.apache.hadoop.fs.Path upperPath, org.apache.hadoop.fs.permission.FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress)
boolean	delete(org.apache.hadoop.fs.Path upperPath, boolean recursive)
org.apache.hadoop.fs.FileStatus[]	dirModeListStatus(org.apache.hadoop.fs.Path upperPath) Implements listing for SfsInodeType#DIR_MODE.
String	getCanonicalServiceName()
org.apache.hadoop.fs.FileStatus	getFileStatus(org.apache.hadoop.fs.Path upperPath)
String	getScheme()
URI	getUri()
org.apache.hadoop.fs.Path	getWorkingDirectory()
void	initialize(URI uri, org.apache.hadoop.conf.Configuration conf)
org.apache.hadoop.fs.FileStatus[]	listStatus(org.apache.hadoop.fs.Path upperPath)
static org.apache.hadoop.fs.FileStatus	makeDirFileStatus(org.apache.hadoop.fs.Path upperPath, org.apache.hadoop.fs.FileStatus status)
org.apache.hadoop.fs.FileStatus	makeDirFileStatus(org.apache.hadoop.fs.Path upperPath, org.apache.hadoop.fs.Path lowerPath)
org.apache.hadoop.fs.FileStatus	makeFileStatus(org.apache.hadoop.fs.Path upperPath, org.apache.hadoop.fs.Path lowerPath)
boolean	mkdirs(org.apache.hadoop.fs.Path upperPath, org.apache.hadoop.fs.permission.FsPermission permission)
org.apache.hadoop.fs.FSDataInputStream	open(org.apache.hadoop.fs.Path upperPath, int bufferSize)
boolean	rename(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst)
void	setWorkingDirectory(org.apache.hadoop.fs.Path new_dir)

Methods inherited from class org.apache.hadoop.fs.FileSystem

access, append, append, appendFile, areSymlinksEnabled, cancelDeleteOnExit, canonicalizeUri, checkPath, clearStatistics, close, closeAll, closeAllForUGI, completeLocalOutput, concat, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createDataInputStreamBuilder, createDataInputStreamBuilder, createDataOutputStreamBuilder, createFile, createMultipartUploader, createNewFile, createNonRecursive, createNonRecursive, createNonRecursive, createPathHandle, createSnapshot, createSnapshot, createSymlink, delete, deleteOnExit, deleteSnapshot, enableSymlinks, exists, fixRelativePart, get, get, get, getAclStatus, getAdditionalTokenIssuers, getAllStatistics, getAllStoragePolicies, getBlockSize, getCanonicalUri, getChildFileSystems, getContentSummary, getDefaultBlockSize, getDefaultBlockSize, getDefaultPort, getDefaultReplication, getDefaultReplication, getDefaultUri, getDelegationToken, getFileBlockLocations, getFileBlockLocations, getFileChecksum, getFileChecksum, getFileLinkStatus, getFileSystemClass, getFSofPath, getGlobalStorageStatistics, getHomeDirectory, getInitialWorkingDirectory, getLength, getLinkTarget, getLocal, getName, getNamed, getPathHandle, getQuotaUsage, getReplication, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getStoragePolicy, getStorageStatistics, getTrashRoot, getTrashRoots, getUsed, getUsed, getXAttr, getXAttrs, getXAttrs, globStatus, globStatus, hasPathCapability, isDirectory, isFile, listCorruptFileBlocks, listFiles, listLocatedStatus, listLocatedStatus, listStatus, listStatus, listStatus, listStatusBatch, listStatusIterator, listXAttrs, makeQualified, mkdirs, mkdirs, modifyAclEntries, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, msync, newInstance, newInstance, newInstance, newInstanceLocal, open, open, open, openFile, openFile, openFileWithOptions, openFileWithOptions, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, processDeleteOnExit, removeAcl, removeAclEntries, removeDefaultAcl, removeXAttr, rename, renameSnapshot, resolveLink, resolvePath, satisfyStoragePolicy, setAcl, setDefaultUri, setDefaultUri, setOwner, setPermission, setQuota, setQuotaByStorageType, setReplication, setStoragePolicy, setTimes, setVerifyChecksum, setWriteChecksum, setXAttr, setXAttr, startLocalOutput, supportsSymlinks, truncate, unsetStoragePolicy

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.security.token.DelegationTokenIssuer

addDelegationTokens, collectDelegationTokens

Constructor Detail

SingleFileSystem

public SingleFileSystem()

Method Detail

getScheme

public String getScheme()

Overrides:

getScheme in class org.apache.hadoop.fs.FileSystem

initialize

public void initialize(URI uri,
                       org.apache.hadoop.conf.Configuration conf)
                throws IOException

Overrides:

initialize in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

getUri

public URI getUri()

Specified by:

getUri in class org.apache.hadoop.fs.FileSystem

open

public org.apache.hadoop.fs.FSDataInputStream open(org.apache.hadoop.fs.Path upperPath,
                                                   int bufferSize)
                                            throws IOException

Specified by:

open in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

getFileStatus

public org.apache.hadoop.fs.FileStatus getFileStatus(org.apache.hadoop.fs.Path upperPath)
                                              throws IOException

Specified by:

getFileStatus in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

listStatus

public org.apache.hadoop.fs.FileStatus[] listStatus(org.apache.hadoop.fs.Path upperPath)
                                             throws FileNotFoundException,
                                                    IOException

Specified by:

listStatus in class org.apache.hadoop.fs.FileSystem

Throws:

FileNotFoundException

IOException

setWorkingDirectory

public void setWorkingDirectory(org.apache.hadoop.fs.Path new_dir)

Specified by:

setWorkingDirectory in class org.apache.hadoop.fs.FileSystem

getWorkingDirectory

public org.apache.hadoop.fs.Path getWorkingDirectory()

Specified by:

getWorkingDirectory in class org.apache.hadoop.fs.FileSystem

create

public org.apache.hadoop.fs.FSDataOutputStream create(org.apache.hadoop.fs.Path upperPath,
                                                      org.apache.hadoop.fs.permission.FsPermission permission,
                                                      boolean overwrite,
                                                      int bufferSize,
                                                      short replication,
                                                      long blockSize,
                                                      org.apache.hadoop.util.Progressable progress)
                                               throws IOException

Specified by:

create in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

append

public org.apache.hadoop.fs.FSDataOutputStream append(org.apache.hadoop.fs.Path upperPath,
                                                      int bufferSize,
                                                      org.apache.hadoop.util.Progressable progress)
                                               throws IOException

Specified by:

append in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

rename

public boolean rename(org.apache.hadoop.fs.Path src,
                      org.apache.hadoop.fs.Path dst)
               throws IOException

Specified by:

rename in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

delete

public boolean delete(org.apache.hadoop.fs.Path upperPath,
                      boolean recursive)
               throws IOException

Specified by:

delete in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

mkdirs

public boolean mkdirs(org.apache.hadoop.fs.Path upperPath,
                      org.apache.hadoop.fs.permission.FsPermission permission)
               throws IOException

Specified by:

mkdirs in class org.apache.hadoop.fs.FileSystem

Throws:

IOException

getCanonicalServiceName

public String getCanonicalServiceName()

Specified by:

getCanonicalServiceName in interface org.apache.hadoop.security.token.DelegationTokenIssuer

Overrides:

getCanonicalServiceName in class org.apache.hadoop.fs.FileSystem

dirModeListStatus

public org.apache.hadoop.fs.FileStatus[] dirModeListStatus(org.apache.hadoop.fs.Path upperPath)
                                                    throws IOException

Implements listing for SfsInodeType#DIR_MODE.

Throws:

IOException

makeFileStatus

public org.apache.hadoop.fs.FileStatus makeFileStatus(org.apache.hadoop.fs.Path upperPath,
                                                      org.apache.hadoop.fs.Path lowerPath)
                                               throws IOException

Throws:

IOException

makeDirFileStatus

public org.apache.hadoop.fs.FileStatus makeDirFileStatus(org.apache.hadoop.fs.Path upperPath,
                                                         org.apache.hadoop.fs.Path lowerPath)
                                                  throws IOException

Throws:

IOException

makeDirFileStatus

public static org.apache.hadoop.fs.FileStatus makeDirFileStatus(org.apache.hadoop.fs.Path upperPath,
                                                                org.apache.hadoop.fs.FileStatus status)
                                                         throws IOException

Throws:

IOException