Package io.trino.plugin.hive.s3

Class TrinoS3FileSystem

  • All Implemented Interfaces:
    Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable, org.apache.hadoop.security.token.DelegationTokenIssuer
    public class TrinoS3FileSystem
extends org.apache.hadoop.fs.FileSystem

    • Nested Class Summary

      • Nested classes/interfaces inherited from class org.apache.hadoop.fs.FileSystem

        org.apache.hadoop.fs.FileSystem.DirectoryEntries, org.apache.hadoop.fs.FileSystem.DirListingIterator<T extends org.apache.hadoop.fs.FileStatus>, org.apache.hadoop.fs.FileSystem.Statistics

    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      org.apache.hadoop.fs.FSDataOutputStream append​(org.apache.hadoop.fs.Path f, int bufferSize, org.apache.hadoop.util.Progressable progress)  
      void close()  
      org.apache.hadoop.fs.FSDataOutputStream create​(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, org.apache.hadoop.util.Progressable progress)  
      boolean delete​(org.apache.hadoop.fs.Path path, boolean recursive)  
      static String extractBucketName​(URI uri)
      Helper function used to work around the fact that if you use an S3 bucket with an '_' that java.net.URI behaves differently and sets the host value to null whereas S3 buckets without '_' have a properly set host field.
      protected String getBucketName​(URI uri)  
      org.apache.hadoop.fs.FileStatus getFileStatus​(org.apache.hadoop.fs.Path path)  
      static TrinoS3FileSystemStats getFileSystemStats()  
      URI getUri()  
      org.apache.hadoop.fs.Path getWorkingDirectory()  
      void initialize​(URI uri, org.apache.hadoop.conf.Configuration conf)  
      static String keyFromPath​(org.apache.hadoop.fs.Path path)  
      org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> listFiles​(org.apache.hadoop.fs.Path path, boolean recursive)  
      org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> listFilesByPrefix​(org.apache.hadoop.fs.Path prefix, boolean recursive)  
      org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> listLocatedStatus​(org.apache.hadoop.fs.Path path)  
      org.apache.hadoop.fs.FileStatus[] listStatus​(org.apache.hadoop.fs.Path path)  
      boolean mkdirs​(org.apache.hadoop.fs.Path f, org.apache.hadoop.fs.permission.FsPermission permission)  
      org.apache.hadoop.fs.FSDataInputStream open​(org.apache.hadoop.fs.Path path, int bufferSize)  
      boolean rename​(org.apache.hadoop.fs.Path src, org.apache.hadoop.fs.Path dst)  
      void setWorkingDirectory​(org.apache.hadoop.fs.Path path)  

      • Methods inherited from class org.apache.hadoop.fs.FileSystem

        access, append, append, appendFile, areSymlinksEnabled, cancelDeleteOnExit, canonicalizeUri, checkPath, clearStatistics, closeAll, closeAllForUGI, completeLocalOutput, concat, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyFromLocalFile, copyToLocalFile, copyToLocalFile, copyToLocalFile, create, create, create, create, create, create, create, create, create, create, create, create, createFile, createNewFile, createNonRecursive, createNonRecursive, createNonRecursive, createPathHandle, createSnapshot, createSnapshot, createSymlink, delete, deleteOnExit, deleteSnapshot, enableSymlinks, exists, fixRelativePart, get, get, get, getAclStatus, getAdditionalTokenIssuers, getAllStatistics, getAllStoragePolicies, getBlockSize, getCanonicalServiceName, getCanonicalUri, getChildFileSystems, getContentSummary, getDefaultBlockSize, getDefaultBlockSize, getDefaultPort, getDefaultReplication, getDefaultReplication, getDefaultUri, getDelegationToken, getFileBlockLocations, getFileBlockLocations, getFileChecksum, getFileChecksum, getFileLinkStatus, getFileSystemClass, getFSofPath, getGlobalStorageStatistics, getHomeDirectory, getInitialWorkingDirectory, getLength, getLinkTarget, getLocal, getName, getNamed, getPathHandle, getQuotaUsage, getReplication, getScheme, getServerDefaults, getServerDefaults, getStatistics, getStatistics, getStatus, getStatus, getStoragePolicy, getStorageStatistics, getTrashRoot, getTrashRoots, getUsed, getUsed, getXAttr, getXAttrs, getXAttrs, globStatus, globStatus, isDirectory, isFile, listCorruptFileBlocks, listLocatedStatus, listStatus, listStatus, listStatus, listStatusBatch, listStatusIterator, listXAttrs, makeQualified, mkdirs, mkdirs, modifyAclEntries, moveFromLocalFile, moveFromLocalFile, moveToLocalFile, newInstance, newInstance, newInstance, newInstanceLocal, open, open, open, primitiveCreate, primitiveMkdir, primitiveMkdir, printStatistics, processDeleteOnExit, removeAcl, removeAclEntries, removeDefaultAcl, removeXAttr, rename, renameSnapshot, resolveLink, resolvePath, setAcl, setDefaultUri, setDefaultUri, setOwner, setPermission, setReplication, setStoragePolicy, setTimes, setVerifyChecksum, setWriteChecksum, setXAttr, setXAttr, startLocalOutput, supportsSymlinks, truncate, unsetStoragePolicy

      • Methods inherited from class org.apache.hadoop.conf.Configured

        getConf, setConf

      • Methods inherited from interface org.apache.hadoop.security.token.DelegationTokenIssuer

        addDelegationTokens

    • Constructor Detail

      • TrinoS3FileSystem

        public TrinoS3FileSystem()

    • Method Detail

      • initialize

        public void initialize​(URI uri,
                       org.apache.hadoop.conf.Configuration conf)
                throws IOException
        Overrides:
        initialize in class org.apache.hadoop.fs.FileSystem
        Throws:
        IOException

      • getUri

        public URI getUri()
        Specified by:
        getUri in class org.apache.hadoop.fs.FileSystem

      • getWorkingDirectory

        public org.apache.hadoop.fs.Path getWorkingDirectory()
        Specified by:
        getWorkingDirectory in class org.apache.hadoop.fs.FileSystem

      • setWorkingDirectory

        public void setWorkingDirectory​(org.apache.hadoop.fs.Path path)
        Specified by:
        setWorkingDirectory in class org.apache.hadoop.fs.FileSystem

      • listStatus

        public org.apache.hadoop.fs.FileStatus[] listStatus​(org.apache.hadoop.fs.Path path)
                                             throws IOException
        Specified by:
        listStatus in class org.apache.hadoop.fs.FileSystem
        Throws:
        IOException

      • listFiles

        public org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> listFiles​(org.apache.hadoop.fs.Path path,
                                                                                             boolean recursive)
        Overrides:
        listFiles in class org.apache.hadoop.fs.FileSystem

      • listFilesByPrefix

        public org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> listFilesByPrefix​(org.apache.hadoop.fs.Path prefix,
                                                                                                     boolean recursive)

      • listLocatedStatus

        public org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> listLocatedStatus​(org.apache.hadoop.fs.Path path)
        Overrides:
        listLocatedStatus in class org.apache.hadoop.fs.FileSystem

      • getFileStatus

        public org.apache.hadoop.fs.FileStatus getFileStatus​(org.apache.hadoop.fs.Path path)
                                              throws IOException
        Specified by:
        getFileStatus in class org.apache.hadoop.fs.FileSystem
        Throws:
        IOException

      • open

        public org.apache.hadoop.fs.FSDataInputStream open​(org.apache.hadoop.fs.Path path,
                                                   int bufferSize)
        Specified by:
        open in class org.apache.hadoop.fs.FileSystem

      • create

        public org.apache.hadoop.fs.FSDataOutputStream create​(org.apache.hadoop.fs.Path path,
                                                      org.apache.hadoop.fs.permission.FsPermission permission,
                                                      boolean overwrite,
                                                      int bufferSize,
                                                      short replication,
                                                      long blockSize,
                                                      org.apache.hadoop.util.Progressable progress)
                                               throws IOException
        Specified by:
        create in class org.apache.hadoop.fs.FileSystem
        Throws:
        IOException

      • append

        public org.apache.hadoop.fs.FSDataOutputStream append​(org.apache.hadoop.fs.Path f,
                                                      int bufferSize,
                                                      org.apache.hadoop.util.Progressable progress)
        Specified by:
        append in class org.apache.hadoop.fs.FileSystem

      • rename

        public boolean rename​(org.apache.hadoop.fs.Path src,
                      org.apache.hadoop.fs.Path dst)
               throws IOException
        Specified by:
        rename in class org.apache.hadoop.fs.FileSystem
        Throws:
        IOException

      • delete

        public boolean delete​(org.apache.hadoop.fs.Path path,
                      boolean recursive)
               throws IOException
        Specified by:
        delete in class org.apache.hadoop.fs.FileSystem
        Throws:
        IOException

      • mkdirs

        public boolean mkdirs​(org.apache.hadoop.fs.Path f,
                      org.apache.hadoop.fs.permission.FsPermission permission)
        Specified by:
        mkdirs in class org.apache.hadoop.fs.FileSystem

      • keyFromPath

        public static String keyFromPath​(org.apache.hadoop.fs.Path path)

      • getBucketName

        protected String getBucketName​(URI uri)

      • extractBucketName

        public static String extractBucketName​(URI uri)
        Helper function used to work around the fact that if you use an S3 bucket with an '_' that java.net.URI behaves differently and sets the host value to null whereas S3 buckets without '_' have a properly set host field. '_' is only allowed in S3 bucket names in us-east-1.
        Parameters:
        uri - The URI from which to extract a host value.
        Returns:
        The host value where uri.getAuthority() is used when uri.getHost() returns null as long as no UserInfo is present.
        Throws:
        IllegalArgumentException - If the bucket cannot be determined from the URI.