com.facebook.presto.hive.s3

Class PrestoS3FileSystem

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.FileSystem

com.facebook.presto.hive.filesystem.ExtendedFileSystem

com.facebook.presto.hive.s3.PrestoS3FileSystem

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable

public class PrestoS3FileSystem
extends ExtendedFileSystem

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	PrestoS3FileSystem.PrestoS3ObjectMetadata

Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem

FileSystem.Statistics

Field Summary

Fields inherited from class org.apache.hadoop.fs.FileSystem

DEFAULT_FS, FS_DEFAULT_NAME_KEY, LOG, SHUTDOWN_HOOK_PRIORITY, statistics

Constructor Summary

Constructors

Constructor and Description
PrestoS3FileSystem()

Method Summary

Modifier and Type	Method and Description
FSDataOutputStream	append(Path f, int bufferSize, org.apache.hadoop.util.Progressable progress)
void	close()
FSDataOutputStream	create(Path path, org.apache.hadoop.fs.permission.FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress)
boolean	delete(Path path, boolean recursive)
static String	getBucketName(URI uri) Helper function used to work around the fact that if you use an S3 bucket with an '_' that java.net.URI behaves differently and sets the host value to null whereas S3 buckets without '_' have a properly set host field.
FileStatus	getFileStatus(Path path)
static PrestoS3FileSystemStats	getFileSystemStats()
static com.amazonaws.metrics.RequestMetricCollector	getMetricsCollector()
String	getScheme()
URI	getUri()
Path	getWorkingDirectory()
void	initialize(URI uri, org.apache.hadoop.conf.Configuration conf)
static String	keyFromPath(Path path)
RemoteIterator<LocatedFileStatus>	listFiles(Path path, boolean recursive)
RemoteIterator<LocatedFileStatus>	listLocatedStatus(Path path)
FileStatus[]	listStatus(Path path)
boolean	mkdirs(Path f, org.apache.hadoop.fs.permission.FsPermission permission)
FSDataInputStream	open(Path path, int bufferSize)
boolean	rename(Path src, Path dst)
static void	setMetricsCollector(com.amazonaws.metrics.RequestMetricCollector customMetricCollector)
void	setWorkingDirectory(Path path)

Methods inherited from class com.facebook.presto.hive.filesystem.ExtendedFileSystem

listDirectory, listFiles, openFile, renameFileAsync

Methods inherited from class org.apache.hadoop.fs.FileSystem

access, addDelegationTokens, append, append, areSymlinksEnabled, cancelDeleteOnExit, canonicalizeUri, checkPath, clearStatistics, closeAll, closeAllForUGI, completeLocalOutput, concat, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createNewFile, createNonRecursive, createNonRecursive, createNonRecursive, createSnapshot, createSnapshot, createSymlink, delete, deleteOnExit, deleteSnapshot, enableSymlinks, exists, fixRelativePart, get, get, get, getAclStatus, getAllStatistics, getBlockSize, getCanonicalServiceName, getCanonicalUri, getChildFileSystems, getContentSummary, getDefaultBlockSize, getDefaultBlockSize, getDefaultPort, getDefaultReplication, getDefaultReplication, getDefaultUri, getDelegationToken, getFileBlockLocations, getFileBlockLocations, getFileChecksum, getFileChecksum, getFileLinkStatus, getFileSystemClass, getFSofPath, getHomeDirectory, getInitialWorkingDirectory, getLength, getLinkTarget, getLocal, getName, getNamed, getReplication, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getUsed, getXAttr, getXAttrs, getXAttrs, globStatus, globStatus, isDirectory, isFile, listCorruptFileBlocks, listLocatedStatus, listStatus, listStatus, listStatus, listStatusIterator, listXAttrs, makeQualified, mkdirs, mkdirs, modifyAclEntries, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, newInstance, newInstance, newInstance, newInstanceLocal, open, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, processDeleteOnExit, removeAcl, removeAclEntries, removeDefaultAcl, removeXAttr, rename, renameSnapshot, resolveLink, resolvePath, setAcl, setDefaultUri, setDefaultUri, setOwner, setPermission, setReplication, setTimes, setVerifyChecksum, setWriteChecksum, setXAttr, setXAttr, startLocalOutput, supportsSymlinks, truncate

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PrestoS3FileSystem

public PrestoS3FileSystem()

Method Detail

initialize

public void initialize(URI uri,
                       org.apache.hadoop.conf.Configuration conf)
                throws IOException

Overrides:

initialize in class FileSystem

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class FileSystem

Throws:

IOException

getUri

public URI getUri()

Specified by:

getUri in class FileSystem

getScheme

public String getScheme()

Overrides:

getScheme in class FileSystem

getWorkingDirectory

public Path getWorkingDirectory()

Specified by:

getWorkingDirectory in class FileSystem

setWorkingDirectory

public void setWorkingDirectory(Path path)

Specified by:

setWorkingDirectory in class FileSystem

listStatus

public FileStatus[] listStatus(Path path)
                        throws IOException

Specified by:

listStatus in class FileSystem

Throws:

IOException

listFiles

public RemoteIterator<LocatedFileStatus> listFiles(Path path,
                                                   boolean recursive)

Overrides:

listFiles in class FileSystem

listLocatedStatus

public RemoteIterator<LocatedFileStatus> listLocatedStatus(Path path)

Overrides:

listLocatedStatus in class FileSystem

getFileStatus

public FileStatus getFileStatus(Path path)
                         throws IOException

Specified by:

getFileStatus in class FileSystem

Throws:

IOException

open

public FSDataInputStream open(Path path,
                              int bufferSize)

Specified by:

open in class FileSystem

create

public FSDataOutputStream create(Path path,
                                 org.apache.hadoop.fs.permission.FsPermission permission,
                                 boolean overwrite,
                                 int bufferSize,
                                 short replication,
                                 long blockSize,
                                 org.apache.hadoop.util.Progressable progress)
                          throws IOException

Specified by:

create in class FileSystem

Throws:

IOException

append

public FSDataOutputStream append(Path f,
                                 int bufferSize,
                                 org.apache.hadoop.util.Progressable progress)

Specified by:

append in class FileSystem

rename

public boolean rename(Path src,
                      Path dst)
               throws IOException

Specified by:

rename in class FileSystem

Throws:

IOException

delete

public boolean delete(Path path,
                      boolean recursive)
               throws IOException

Specified by:

delete in class FileSystem

Throws:

IOException

mkdirs

public boolean mkdirs(Path f,
                      org.apache.hadoop.fs.permission.FsPermission permission)

Specified by:

mkdirs in class FileSystem

keyFromPath

public static String keyFromPath(Path path)

getBucketName

public static String getBucketName(URI uri)

Helper function used to work around the fact that if you use an S3 bucket with an '_' that java.net.URI behaves differently and sets the host value to null whereas S3 buckets without '_' have a properly set host field. '_' is only allowed in S3 bucket names in us-east-1.

Parameters:

uri - The URI from which to extract a host value.

Returns:

The host value where uri.getAuthority() is used when uri.getHost() returns null as long as no UserInfo is present.

Throws:

IllegalArgumentException - If the bucket can not be determined from the URI.

getFileSystemStats

public static PrestoS3FileSystemStats getFileSystemStats()

getMetricsCollector

public static com.amazonaws.metrics.RequestMetricCollector getMetricsCollector()

setMetricsCollector

public static void setMetricsCollector(com.amazonaws.metrics.RequestMetricCollector customMetricCollector)