org.apache.hadoop.fs.s3a

Interface S3AStore

All Superinterfaces:

AutoCloseable, ClientManager, Closeable, org.apache.hadoop.fs.statistics.IOStatisticsSource

All Known Implementing Classes:

S3AStoreImpl

@InterfaceAudience.LimitedPrivate(value="Extensions")
 @InterfaceStability.Unstable
public interface S3AStore
extends org.apache.hadoop.fs.statistics.IOStatisticsSource, ClientManager

Interface for the S3A Store; S3 client interactions should be via this; mocking is possible for unit tests.

The ClientManager interface is used to create the AWS clients; the base implementation forwards to the implementation of this interface passed in at construction time.

Method Summary

Modifier and Type	Method and Description
Duration	acquireReadCapacity(int capacity) Acquire read capacity for operations.
Duration	acquireWriteCapacity(int capacity) Acquire write capacity for operations.
ClientManager	clientManager()
software.amazon.awssdk.services.s3.model.CompleteMultipartUploadResponse	completeMultipartUpload(software.amazon.awssdk.services.s3.model.CompleteMultipartUploadRequest request) Complete a multipart upload.
Map.Entry<Duration,Optional<software.amazon.awssdk.services.s3.model.DeleteObjectResponse>>	deleteObject(software.amazon.awssdk.services.s3.model.DeleteObjectRequest request) Delete an object.
Map.Entry<Duration,software.amazon.awssdk.services.s3.model.DeleteObjectsResponse>	deleteObjects(software.amazon.awssdk.services.s3.model.DeleteObjectsRequest deleteRequest) Perform a bulk object delete operation against S3.
org.apache.hadoop.fs.statistics.DurationTrackerFactory	getDurationTrackerFactory()
RequestFactory	getRequestFactory()
S3AStatisticsContext	getStatisticsContext()
StoreContext	getStoreContext()
void	incrementPutCompletedStatistics(boolean success, long bytes) At the end of a put/multipart upload operation, update the relevant counters and gauges.
void	incrementPutProgressStatistics(String key, long bytes) Callback for use in progress callbacks from put/multipart upload events.
void	incrementPutStartStatistics(long bytes) At the start of a put/multipart upload operation, update the relevant counters.
void	incrementReadOperations() Increment read operations.
void	incrementWriteOperations() Increment the write operation counter.
org.apache.hadoop.fs.statistics.DurationTrackerFactory	nonNullDurationTrackerFactory(org.apache.hadoop.fs.statistics.DurationTrackerFactory factory) Given a possibly null duration tracker factory, return a non-null one for use in tracking durations -either that or the FS tracker itself.
UploadInfo	putObject(software.amazon.awssdk.services.s3.model.PutObjectRequest putObjectRequest, File file, ProgressableProgressListener listener) Start a transfer-manager managed async PUT of an object, incrementing the put requests and put bytes counters.
software.amazon.awssdk.services.s3.model.UploadPartResponse	uploadPart(software.amazon.awssdk.services.s3.model.UploadPartRequest request, software.amazon.awssdk.core.sync.RequestBody body, org.apache.hadoop.fs.statistics.DurationTrackerFactory durationTrackerFactory) Upload part of a multi-partition file.
software.amazon.awssdk.transfer.s3.model.CompletedFileUpload	waitForUploadCompletion(String key, UploadInfo uploadInfo) Wait for an upload to complete.

Methods inherited from interface org.apache.hadoop.fs.statistics.IOStatisticsSource

getIOStatistics

Methods inherited from interface org.apache.hadoop.fs.s3a.impl.ClientManager

close, getOrCreateAsyncClient, getOrCreateAsyncS3ClientUnchecked, getOrCreateS3Client, getOrCreateS3ClientUnchecked, getOrCreateTransferManager

Method Detail

acquireWriteCapacity

Duration acquireWriteCapacity(int capacity)

Acquire write capacity for operations. This should be done within retry loops.

Parameters:

capacity - capacity to acquire.

Returns:

time spent waiting for output.

acquireReadCapacity

Duration acquireReadCapacity(int capacity)

Acquire read capacity for operations. This should be done within retry loops.

Parameters:

capacity - capacity to acquire.

Returns:

time spent waiting for output.

getStoreContext

StoreContext getStoreContext()

getDurationTrackerFactory

org.apache.hadoop.fs.statistics.DurationTrackerFactory getDurationTrackerFactory()

getStatisticsContext

S3AStatisticsContext getStatisticsContext()

getRequestFactory

RequestFactory getRequestFactory()

clientManager

ClientManager clientManager()

incrementReadOperations

void incrementReadOperations()

Increment read operations.

incrementWriteOperations

void incrementWriteOperations()

Increment the write operation counter. This is somewhat inaccurate, as it appears to be invoked more often than needed in progress callbacks.

incrementPutStartStatistics

void incrementPutStartStatistics(long bytes)

At the start of a put/multipart upload operation, update the relevant counters.

Parameters:

bytes - bytes in the request.

incrementPutCompletedStatistics

void incrementPutCompletedStatistics(boolean success,
                                     long bytes)

At the end of a put/multipart upload operation, update the relevant counters and gauges.

Parameters:

success - did the operation succeed?

bytes - bytes in the request.

incrementPutProgressStatistics

void incrementPutProgressStatistics(String key,
                                    long bytes)

Callback for use in progress callbacks from put/multipart upload events. Increments those statistics which are expected to be updated during the ongoing upload operation.

Parameters:

key - key to file that is being written (for logging)

bytes - bytes successfully uploaded.

nonNullDurationTrackerFactory

org.apache.hadoop.fs.statistics.DurationTrackerFactory nonNullDurationTrackerFactory(org.apache.hadoop.fs.statistics.DurationTrackerFactory factory)

Given a possibly null duration tracker factory, return a non-null one for use in tracking durations -either that or the FS tracker itself.

Parameters:

factory - factory.

Returns:

a non-null factory.

deleteObjects

@Retries.RetryRaw
Map.Entry<Duration,software.amazon.awssdk.services.s3.model.DeleteObjectsResponse> deleteObjects(software.amazon.awssdk.services.s3.model.DeleteObjectsRequest deleteRequest)
                                                                                                            throws MultiObjectDeleteException,
                                                                                                                   software.amazon.awssdk.core.exception.SdkException,
                                                                                                                   IOException

Perform a bulk object delete operation against S3. Increments the OBJECT_DELETE_REQUESTS and write operation statistics

OBJECT_DELETE_OBJECTS is updated with the actual number of objects deleted in the request.

Retry policy: retry untranslated; delete considered idempotent. If the request is throttled, this is logged in the throttle statistics, with the counter set to the number of keys, rather than the number of invocations of the delete operation. This is because S3 considers each key as one mutating operation on the store when updating its load counters on a specific partition of an S3 bucket. If only the request was measured, this operation would under-report. A write capacity will be requested proportional to the number of keys preset in the request and will be re-requested during retries such that retries throttle better. If the request is throttled, the time spent is recorded in a duration IOStat named STORE_IO_RATE_LIMITED_DURATION.

Parameters:

deleteRequest - keys to delete on the s3-backend

Returns:

the AWS response

Throws:

MultiObjectDeleteException - one or more of the keys could not be deleted.

software.amazon.awssdk.core.exception.SdkException - amazon-layer failure.

IOException - IO problems.

deleteObject

@Retries.RetryRaw
Map.Entry<Duration,Optional<software.amazon.awssdk.services.s3.model.DeleteObjectResponse>> deleteObject(software.amazon.awssdk.services.s3.model.DeleteObjectRequest request)
                                                                                                                    throws software.amazon.awssdk.core.exception.SdkException

Delete an object. Increments the OBJECT_DELETE_REQUESTS statistics.

Retry policy: retry untranslated; delete considered idempotent. 404 errors other than bucket not found are swallowed; this can be raised by third party stores (GCS).

A write capacity of 1 ( as it is signle object delete) will be requested before the delete call and will be re-requested during retries such that retries throttle better. If the request is throttled, the time spent is recorded in a duration IOStat named STORE_IO_RATE_LIMITED_DURATION. If an exception is caught and swallowed, the response will be empty; otherwise it will be the response from the delete operation.

Parameters:

request - request to make

Returns:

the total duration and response.

Throws:

software.amazon.awssdk.core.exception.SdkException - problems working with S3

IllegalArgumentException - if the request was rejected due to a mistaken attempt to delete the root directory.

uploadPart

@Retries.OnceRaw
software.amazon.awssdk.services.s3.model.UploadPartResponse uploadPart(software.amazon.awssdk.services.s3.model.UploadPartRequest request,
                                                                                        software.amazon.awssdk.core.sync.RequestBody body,
                                                                                        org.apache.hadoop.fs.statistics.DurationTrackerFactory durationTrackerFactory)
                                                                                 throws software.amazon.awssdk.awscore.exception.AwsServiceException,
                                                                                        UncheckedIOException

Upload part of a multi-partition file. Increments the write and put counters. Important: this call does not close any input stream in the body.

Retry Policy: none.

Parameters:

durationTrackerFactory - duration tracker factory for operation

request - the upload part request.

body - the request body.

Returns:

the result of the operation.

Throws:

software.amazon.awssdk.awscore.exception.AwsServiceException - on problems

UncheckedIOException - failure to instantiate the s3 client

putObject

@Retries.OnceRaw
UploadInfo putObject(software.amazon.awssdk.services.s3.model.PutObjectRequest putObjectRequest,
                                      File file,
                                      ProgressableProgressListener listener)
                               throws IOException

Start a transfer-manager managed async PUT of an object, incrementing the put requests and put bytes counters.

It does not update the other counters, as existing code does that as progress callbacks come in. Byte length is calculated from the file length, or, if there is no file, from the content length of the header.

Because the operation is async, any stream supplied in the request must reference data (files, buffers) which stay valid until the upload completes. Retry policy: N/A: the transfer manager is performing the upload. Auditing: must be inside an audit span.

Parameters:

putObjectRequest - the request

file - the file to be uploaded

listener - the progress listener for the request

Returns:

the upload initiated

Throws:

IOException - if transfer manager creation failed.

waitForUploadCompletion

@Retries.OnceTranslated
software.amazon.awssdk.transfer.s3.model.CompletedFileUpload waitForUploadCompletion(String key,
                                                                                                             UploadInfo uploadInfo)
                                                                                                      throws IOException

Wait for an upload to complete. If the upload (or its result collection) failed, this is where the failure is raised as an AWS exception. Calls incrementPutCompletedStatistics(boolean, long) to update the statistics.

Parameters:

key - destination key

uploadInfo - upload to wait for

Returns:

the upload result

Throws:

IOException - IO failure

CancellationException - if the wait() was cancelled

completeMultipartUpload

@Retries.OnceRaw
software.amazon.awssdk.services.s3.model.CompleteMultipartUploadResponse completeMultipartUpload(software.amazon.awssdk.services.s3.model.CompleteMultipartUploadRequest request)

Complete a multipart upload.

Parameters:

request - request

Returns:

the response