org.apache.iceberg.aws.s3

Class S3FileIO

java.lang.Object

org.apache.iceberg.aws.s3.S3FileIO

All Implemented Interfaces:

java.io.Closeable, java.io.Serializable, java.lang.AutoCloseable, org.apache.iceberg.io.CredentialSupplier, org.apache.iceberg.io.FileIO, org.apache.iceberg.io.SupportsBulkOperations, org.apache.iceberg.io.SupportsPrefixOperations

public class S3FileIO
extends java.lang.Object
implements org.apache.iceberg.io.FileIO, org.apache.iceberg.io.SupportsBulkOperations, org.apache.iceberg.io.SupportsPrefixOperations, org.apache.iceberg.io.CredentialSupplier

FileIO implementation backed by S3.

Locations used must follow the conventions for S3 URIs (e.g. s3://bucket/path...). URIs with schemes s3a, s3n, https are also treated as s3 file paths. Using this FileIO with other schemes will result in ValidationException.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
S3FileIO() No-arg constructor to load the FileIO dynamically.
S3FileIO(org.apache.iceberg.util.SerializableSupplier<software.amazon.awssdk.services.s3.S3Client> s3) Constructor with custom s3 supplier and default AWS properties.
S3FileIO(org.apache.iceberg.util.SerializableSupplier<software.amazon.awssdk.services.s3.S3Client> s3, AwsProperties awsProperties) Constructor with custom s3 supplier and AWS properties.

Method Summary

Modifier and Type	Method and Description
void	close()
void	deleteFile(java.lang.String path)
void	deleteFiles(java.lang.Iterable<java.lang.String> paths) Deletes the given paths in a batched manner.
void	deletePrefix(java.lang.String prefix) This method provides a "best-effort" to delete all objects under the given prefix.
java.lang.String	getCredential()
void	initialize(java.util.Map<java.lang.String,java.lang.String> props)
java.lang.Iterable<org.apache.iceberg.io.FileInfo>	listPrefix(java.lang.String prefix)
org.apache.iceberg.io.InputFile	newInputFile(java.lang.String path)
org.apache.iceberg.io.InputFile	newInputFile(java.lang.String path, long length)
org.apache.iceberg.io.OutputFile	newOutputFile(java.lang.String path)
java.util.Map<java.lang.String,java.lang.String>	properties()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.iceberg.io.FileIO

deleteFile, deleteFile

Constructor Detail

S3FileIO

public S3FileIO()

No-arg constructor to load the FileIO dynamically.

All fields are initialized by calling initialize(Map) later.

S3FileIO

public S3FileIO(org.apache.iceberg.util.SerializableSupplier<software.amazon.awssdk.services.s3.S3Client> s3)

Constructor with custom s3 supplier and default AWS properties.

Calling initialize(Map) will overwrite information set in this constructor.

Parameters:

s3 - s3 supplier

S3FileIO

public S3FileIO(org.apache.iceberg.util.SerializableSupplier<software.amazon.awssdk.services.s3.S3Client> s3,
                AwsProperties awsProperties)

Constructor with custom s3 supplier and AWS properties.

Calling initialize(Map) will overwrite information set in this constructor.

Parameters:

s3 - s3 supplier

awsProperties - aws properties

Method Detail

newInputFile

public org.apache.iceberg.io.InputFile newInputFile(java.lang.String path)

Specified by:

newInputFile in interface org.apache.iceberg.io.FileIO

newInputFile

public org.apache.iceberg.io.InputFile newInputFile(java.lang.String path,
                                                    long length)

Specified by:

newInputFile in interface org.apache.iceberg.io.FileIO

newOutputFile

public org.apache.iceberg.io.OutputFile newOutputFile(java.lang.String path)

Specified by:

newOutputFile in interface org.apache.iceberg.io.FileIO

deleteFile

public void deleteFile(java.lang.String path)

Specified by:

deleteFile in interface org.apache.iceberg.io.FileIO

properties

public java.util.Map<java.lang.String,java.lang.String> properties()

Specified by:

properties in interface org.apache.iceberg.io.FileIO

deleteFiles

public void deleteFiles(java.lang.Iterable<java.lang.String> paths)
                 throws org.apache.iceberg.io.BulkDeletionFailureException

Deletes the given paths in a batched manner.

The paths are grouped by bucket, and deletion is triggered when we either reach the configured batch size or have a final remainder batch for each bucket.

Specified by:

deleteFiles in interface org.apache.iceberg.io.SupportsBulkOperations

Parameters:

paths - paths to delete

Throws:

org.apache.iceberg.io.BulkDeletionFailureException

listPrefix

public java.lang.Iterable<org.apache.iceberg.io.FileInfo> listPrefix(java.lang.String prefix)

Specified by:

listPrefix in interface org.apache.iceberg.io.SupportsPrefixOperations

deletePrefix

public void deletePrefix(java.lang.String prefix)

This method provides a "best-effort" to delete all objects under the given prefix. Bulk delete operations are used and no reattempt is made for deletes if they fail, but will log any individual objects that are not deleted as part of the bulk operation.

Specified by:

deletePrefix in interface org.apache.iceberg.io.SupportsPrefixOperations

Parameters:

prefix - prefix to delete

getCredential

public java.lang.String getCredential()

Specified by:

getCredential in interface org.apache.iceberg.io.CredentialSupplier

initialize

public void initialize(java.util.Map<java.lang.String,java.lang.String> props)

Specified by:

initialize in interface org.apache.iceberg.io.FileIO

close

public void close()

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface org.apache.iceberg.io.FileIO