Package org.apache.iceberg.gcp.gcs

Class GCSFileIO

  • All Implemented Interfaces:
    java.io.Closeable, java.io.Serializable, java.lang.AutoCloseable, org.apache.iceberg.io.DelegateFileIO, org.apache.iceberg.io.FileIO, org.apache.iceberg.io.SupportsBulkOperations, org.apache.iceberg.io.SupportsPrefixOperations
    public class GCSFileIO
extends java.lang.Object
implements org.apache.iceberg.io.DelegateFileIO
    FileIO Implementation backed by Google Cloud Storage (GCS)

    Locations follow the conventions used by BlobId.fromGsUtilUri that follow the convention 

    gs://<bucket>/<blob_path>

    See Cloud Storage Overview

    See Also:
    Serialized Form

    • Constructor Summary

      Constructors 
      Constructor Description
      GCSFileIO()
      No-arg constructor to load the FileIO dynamically.
      GCSFileIO​(org.apache.iceberg.util.SerializableSupplier<com.google.cloud.storage.Storage> storageSupplier, GCPProperties gcpProperties)
      Constructor with custom storage supplier and GCP properties.

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      com.google.cloud.storage.Storage client()  
      void close()  
      void deleteFile​(java.lang.String path)  
      void deleteFiles​(java.lang.Iterable<java.lang.String> pathsToDelete)  
      void deletePrefix​(java.lang.String prefix)  
      void initialize​(java.util.Map<java.lang.String,​java.lang.String> props)  
      java.lang.Iterable<org.apache.iceberg.io.FileInfo> listPrefix​(java.lang.String prefix)  
      org.apache.iceberg.io.InputFile newInputFile​(java.lang.String path)  
      org.apache.iceberg.io.InputFile newInputFile​(java.lang.String path, long length)  
      org.apache.iceberg.io.OutputFile newOutputFile​(java.lang.String path)  
      java.util.Map<java.lang.String,​java.lang.String> properties()  

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

      • Methods inherited from interface org.apache.iceberg.io.FileIO

        deleteFile, deleteFile, newInputFile, newInputFile, newInputFile

    • Constructor Detail

      • GCSFileIO

        public GCSFileIO()
        No-arg constructor to load the FileIO dynamically.

        All fields are initialized by calling initialize(Map) later.

      • GCSFileIO

        public GCSFileIO​(org.apache.iceberg.util.SerializableSupplier<com.google.cloud.storage.Storage> storageSupplier,
                 GCPProperties gcpProperties)
        Constructor with custom storage supplier and GCP properties.

        Calling initialize(Map) will overwrite information set in this constructor.

        Parameters:
        storageSupplier - storage supplier
        gcpProperties - gcp properties

    • Method Detail

      • newInputFile

        public org.apache.iceberg.io.InputFile newInputFile​(java.lang.String path)
        Specified by:
        newInputFile in interface org.apache.iceberg.io.FileIO

      • newInputFile

        public org.apache.iceberg.io.InputFile newInputFile​(java.lang.String path,
                                                    long length)
        Specified by:
        newInputFile in interface org.apache.iceberg.io.FileIO

      • newOutputFile

        public org.apache.iceberg.io.OutputFile newOutputFile​(java.lang.String path)
        Specified by:
        newOutputFile in interface org.apache.iceberg.io.FileIO

      • deleteFile

        public void deleteFile​(java.lang.String path)
        Specified by:
        deleteFile in interface org.apache.iceberg.io.FileIO

      • properties

        public java.util.Map<java.lang.String,​java.lang.String> properties()
        Specified by:
        properties in interface org.apache.iceberg.io.FileIO

      • client

        public com.google.cloud.storage.Storage client()

      • initialize

        public void initialize​(java.util.Map<java.lang.String,​java.lang.String> props)
        Specified by:
        initialize in interface org.apache.iceberg.io.FileIO

      • close

        public void close()
        Specified by:
        close in interface java.lang.AutoCloseable
        Specified by:
        close in interface java.io.Closeable
        Specified by:
        close in interface org.apache.iceberg.io.FileIO

      • listPrefix

        public java.lang.Iterable<org.apache.iceberg.io.FileInfo> listPrefix​(java.lang.String prefix)
        Specified by:
        listPrefix in interface org.apache.iceberg.io.SupportsPrefixOperations

      • deletePrefix

        public void deletePrefix​(java.lang.String prefix)
        Specified by:
        deletePrefix in interface org.apache.iceberg.io.SupportsPrefixOperations

      • deleteFiles

        public void deleteFiles​(java.lang.Iterable<java.lang.String> pathsToDelete)
                 throws org.apache.iceberg.io.BulkDeletionFailureException
        Specified by:
        deleteFiles in interface org.apache.iceberg.io.SupportsBulkOperations
        Throws:
        org.apache.iceberg.io.BulkDeletionFailureException