Package org.apache.iceberg.aws.s3

Class S3FileIO

  • All Implemented Interfaces:
    java.io.Closeable, java.io.Serializable, java.lang.AutoCloseable, org.apache.iceberg.io.CredentialSupplier, org.apache.iceberg.io.DelegateFileIO, org.apache.iceberg.io.FileIO, org.apache.iceberg.io.SupportsBulkOperations, org.apache.iceberg.io.SupportsPrefixOperations, org.apache.iceberg.io.SupportsRecoveryOperations
    public class S3FileIO
extends java.lang.Object
implements org.apache.iceberg.io.CredentialSupplier, org.apache.iceberg.io.DelegateFileIO, org.apache.iceberg.io.SupportsRecoveryOperations
    FileIO implementation backed by S3.

    Locations used must follow the conventions for S3 URIs (e.g. s3://bucket/path...). URIs with schemes s3a, s3n, https are also treated as s3 file paths. Using this FileIO with other schemes will result in ValidationException.

    See Also:
    Serialized Form

    • Constructor Summary

      Constructors 
      Constructor Description
      S3FileIO()
      No-arg constructor to load the FileIO dynamically.
      S3FileIO​(org.apache.iceberg.util.SerializableSupplier<software.amazon.awssdk.services.s3.S3Client> s3)
      Constructor with custom s3 supplier and S3FileIO properties.
      S3FileIO​(org.apache.iceberg.util.SerializableSupplier<software.amazon.awssdk.services.s3.S3Client> s3, S3FileIOProperties s3FileIOProperties)
      Constructor with custom s3 supplier and S3FileIO properties.

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      software.amazon.awssdk.services.s3.S3Client client()  
      void close()  
      void deleteFile​(java.lang.String path)  
      void deleteFiles​(java.lang.Iterable<java.lang.String> paths)
      Deletes the given paths in a batched manner.
      void deletePrefix​(java.lang.String prefix)
      This method provides a "best-effort" to delete all objects under the given prefix.
      protected void finalize()  
      java.lang.String getCredential()  
      void initialize​(java.util.Map<java.lang.String,​java.lang.String> props)  
      java.lang.Iterable<org.apache.iceberg.io.FileInfo> listPrefix​(java.lang.String prefix)  
      org.apache.iceberg.io.InputFile newInputFile​(java.lang.String path)  
      org.apache.iceberg.io.InputFile newInputFile​(java.lang.String path, long length)  
      org.apache.iceberg.io.OutputFile newOutputFile​(java.lang.String path)  
      java.util.Map<java.lang.String,​java.lang.String> properties()  
      boolean recoverFile​(java.lang.String path)  

      • Methods inherited from class java.lang.Object

        clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

      • Methods inherited from interface org.apache.iceberg.io.FileIO

        deleteFile, deleteFile, newInputFile, newInputFile, newInputFile

    • Constructor Detail

      • S3FileIO

        public S3FileIO()
        No-arg constructor to load the FileIO dynamically.

        All fields are initialized by calling initialize(Map) later.

      • S3FileIO

        public S3FileIO​(org.apache.iceberg.util.SerializableSupplier<software.amazon.awssdk.services.s3.S3Client> s3)
        Constructor with custom s3 supplier and S3FileIO properties.

        Calling initialize(Map) will overwrite information set in this constructor.

        Parameters:
        s3 - s3 supplier

      • S3FileIO

        public S3FileIO​(org.apache.iceberg.util.SerializableSupplier<software.amazon.awssdk.services.s3.S3Client> s3,
                S3FileIOProperties s3FileIOProperties)
        Constructor with custom s3 supplier and S3FileIO properties.

        Calling initialize(Map) will overwrite information set in this constructor.

        Parameters:
        s3 - s3 supplier
        s3FileIOProperties - S3 FileIO properties

    • Method Detail

      • newInputFile

        public org.apache.iceberg.io.InputFile newInputFile​(java.lang.String path)
        Specified by:
        newInputFile in interface org.apache.iceberg.io.FileIO

      • newInputFile

        public org.apache.iceberg.io.InputFile newInputFile​(java.lang.String path,
                                                    long length)
        Specified by:
        newInputFile in interface org.apache.iceberg.io.FileIO

      • newOutputFile

        public org.apache.iceberg.io.OutputFile newOutputFile​(java.lang.String path)
        Specified by:
        newOutputFile in interface org.apache.iceberg.io.FileIO

      • deleteFile

        public void deleteFile​(java.lang.String path)
        Specified by:
        deleteFile in interface org.apache.iceberg.io.FileIO

      • properties

        public java.util.Map<java.lang.String,​java.lang.String> properties()
        Specified by:
        properties in interface org.apache.iceberg.io.FileIO

      • deleteFiles

        public void deleteFiles​(java.lang.Iterable<java.lang.String> paths)
                 throws org.apache.iceberg.io.BulkDeletionFailureException
        Deletes the given paths in a batched manner.

        The paths are grouped by bucket, and deletion is triggered when we either reach the configured batch size or have a final remainder batch for each bucket.

        Specified by:
        deleteFiles in interface org.apache.iceberg.io.SupportsBulkOperations
        Parameters:
        paths - paths to delete
        Throws:
        org.apache.iceberg.io.BulkDeletionFailureException

      • listPrefix

        public java.lang.Iterable<org.apache.iceberg.io.FileInfo> listPrefix​(java.lang.String prefix)
        Specified by:
        listPrefix in interface org.apache.iceberg.io.SupportsPrefixOperations

      • deletePrefix

        public void deletePrefix​(java.lang.String prefix)
        This method provides a "best-effort" to delete all objects under the given prefix.

        Bulk delete operations are used and no reattempt is made for deletes if they fail, but will log any individual objects that are not deleted as part of the bulk operation.

        Specified by:
        deletePrefix in interface org.apache.iceberg.io.SupportsPrefixOperations
        Parameters:
        prefix - prefix to delete

      • client

        public software.amazon.awssdk.services.s3.S3Client client()

      • getCredential

        public java.lang.String getCredential()
        Specified by:
        getCredential in interface org.apache.iceberg.io.CredentialSupplier

      • initialize

        public void initialize​(java.util.Map<java.lang.String,​java.lang.String> props)
        Specified by:
        initialize in interface org.apache.iceberg.io.FileIO

      • close

        public void close()
        Specified by:
        close in interface java.lang.AutoCloseable
        Specified by:
        close in interface java.io.Closeable
        Specified by:
        close in interface org.apache.iceberg.io.FileIO

      • finalize

        protected void finalize()
                 throws java.lang.Throwable
        Overrides:
        finalize in class java.lang.Object
        Throws:
        java.lang.Throwable

      • recoverFile

        public boolean recoverFile​(java.lang.String path)
        Specified by:
        recoverFile in interface org.apache.iceberg.io.SupportsRecoveryOperations