org.apache.hadoop.fs.s3a

Interface S3AStore

All Superinterfaces:

AutoCloseable, ClientManager, Closeable, org.apache.hadoop.fs.statistics.IOStatisticsSource, ObjectInputStreamFactory, org.apache.hadoop.fs.PathCapabilities, org.apache.hadoop.service.Service, org.apache.hadoop.fs.StreamCapabilities

All Known Implementing Classes:

S3AStoreImpl

@InterfaceAudience.LimitedPrivate(value="Extensions")
 @InterfaceStability.Unstable
public interface S3AStore
extends ClientManager, org.apache.hadoop.fs.statistics.IOStatisticsSource, ObjectInputStreamFactory, org.apache.hadoop.fs.PathCapabilities, org.apache.hadoop.service.Service

Interface for the S3A Store; S3 client interactions should be via this; mocking is possible for unit tests.

The ClientManager interface is used to create the AWS clients; the base implementation forwards to the implementation of this interface passed in at construction time.

The interface extends the Hadoop Service interface and follows its lifecycle: it MUST NOT be used until Service.init(Configuration) has been invoked.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.fs.s3a.impl.streams.ObjectInputStreamFactory

ObjectInputStreamFactory.StreamFactoryCallbacks

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

org.apache.hadoop.service.Service.STATE

Nested classes/interfaces inherited from interface org.apache.hadoop.fs.StreamCapabilities

org.apache.hadoop.fs.StreamCapabilities.StreamCapability

Field Summary

Fields inherited from interface org.apache.hadoop.fs.StreamCapabilities

ABORTABLE_STREAM, DROPBEHIND, HFLUSH, HSYNC, IOSTATISTICS, IOSTATISTICS_CONTEXT, PREADBYTEBUFFER, READAHEAD, READBYTEBUFFER, UNBUFFER, VECTOREDIO, VECTOREDIO_BUFFERS_SLICED

Method Summary

Modifier and Type	Method and Description
Duration	acquireReadCapacity(int capacity) Acquire read capacity for operations.
Duration	acquireWriteCapacity(int capacity) Acquire write capacity for operations.
ClientManager	clientManager()
software.amazon.awssdk.services.s3.model.CompleteMultipartUploadResponse	completeMultipartUpload(software.amazon.awssdk.services.s3.model.CompleteMultipartUploadRequest request) Complete a multipart upload.
File	createTemporaryFileForWriting(String pathStr, long size, org.apache.hadoop.conf.Configuration conf) Demand create the directory allocator, then create a temporary file.
Map.Entry<Duration,Optional<software.amazon.awssdk.services.s3.model.DeleteObjectResponse>>	deleteObject(software.amazon.awssdk.services.s3.model.DeleteObjectRequest request) Delete an object.
Map.Entry<Duration,software.amazon.awssdk.services.s3.model.DeleteObjectsResponse>	deleteObjects(software.amazon.awssdk.services.s3.model.DeleteObjectsRequest deleteRequest) Perform a bulk object delete operation against S3.
org.apache.hadoop.fs.LocalDirAllocator	getDirectoryAllocator() Get the directory allocator.
org.apache.hadoop.fs.statistics.DurationTrackerFactory	getDurationTrackerFactory()
software.amazon.awssdk.core.ResponseInputStream<software.amazon.awssdk.services.s3.model.GetObjectResponse>	getRangedS3Object(String key, long start, long end) Retrieves a specific byte range of an S3 object as a stream.
RequestFactory	getRequestFactory()
S3AStatisticsContext	getStatisticsContext()
StoreContext	getStoreContext()
default boolean	hasCapability(String capability) The StreamCapabilities is part of ObjectInputStreamFactory.
software.amazon.awssdk.services.s3.model.HeadObjectResponse	headObject(String key, ChangeTracker changeTracker, Invoker changeInvoker, S3AFileSystemOperations fsHandler, String operation) Performs a HEAD request on an S3 object to retrieve its metadata.
void	incrementPutCompletedStatistics(boolean success, long bytes) At the end of a put/multipart upload operation, update the relevant counters and gauges.
void	incrementPutProgressStatistics(String key, long bytes) Callback for use in progress callbacks from put/multipart upload events.
void	incrementPutStartStatistics(long bytes) At the start of a put/multipart upload operation, update the relevant counters.
void	incrementReadOperations() Increment read operations.
void	incrementWriteOperations() Increment the write operation counter.
boolean	inputStreamHasCapability(String capability) Return the capabilities of input streams created through the store.
org.apache.hadoop.fs.statistics.DurationTrackerFactory	nonNullDurationTrackerFactory(org.apache.hadoop.fs.statistics.DurationTrackerFactory factory) Given a possibly null duration tracker factory, return a non-null one for use in tracking durations -either that or the FS tracker itself.
UploadInfo	putObject(software.amazon.awssdk.services.s3.model.PutObjectRequest putObjectRequest, File file, ProgressableProgressListener listener) Start a transfer-manager managed async PUT of an object, incrementing the put requests and put bytes counters.
software.amazon.awssdk.services.s3.model.UploadPartResponse	uploadPart(software.amazon.awssdk.services.s3.model.UploadPartRequest request, software.amazon.awssdk.core.sync.RequestBody body, org.apache.hadoop.fs.statistics.DurationTrackerFactory durationTrackerFactory) Upload part of a multi-partition file.
software.amazon.awssdk.transfer.s3.model.CompletedFileUpload	waitForUploadCompletion(String key, UploadInfo uploadInfo) Wait for an upload to complete.

Methods inherited from interface org.apache.hadoop.fs.s3a.impl.ClientManager

getOrCreateAsyncClient, getOrCreateAsyncS3ClientUnchecked, getOrCreateS3Client, getOrCreateS3ClientUnchecked, getOrCreateTransferManager, getOrCreateUnencryptedS3Client

Methods inherited from interface org.apache.hadoop.fs.statistics.IOStatisticsSource

getIOStatistics

Methods inherited from interface org.apache.hadoop.fs.s3a.impl.streams.ObjectInputStreamFactory

bind, factoryRequirements, readObject, streamType

Methods inherited from interface org.apache.hadoop.service.Service

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, registerServiceListener, start, stop, unregisterServiceListener, waitForServiceToStop

Methods inherited from interface org.apache.hadoop.fs.PathCapabilities

hasPathCapability

Method Detail

acquireWriteCapacity

Duration acquireWriteCapacity(int capacity)

Acquire write capacity for operations. This should be done within retry loops.

Parameters:

capacity - capacity to acquire.

Returns:

time spent waiting for output.

acquireReadCapacity

Duration acquireReadCapacity(int capacity)

Acquire read capacity for operations. This should be done within retry loops.

Parameters:

capacity - capacity to acquire.

Returns:

time spent waiting for output.

getStoreContext

StoreContext getStoreContext()

getDurationTrackerFactory

org.apache.hadoop.fs.statistics.DurationTrackerFactory getDurationTrackerFactory()

getStatisticsContext

S3AStatisticsContext getStatisticsContext()

getRequestFactory

RequestFactory getRequestFactory()

clientManager

ClientManager clientManager()

incrementReadOperations

void incrementReadOperations()

Increment read operations.

incrementWriteOperations

void incrementWriteOperations()

Increment the write operation counter. This is somewhat inaccurate, as it appears to be invoked more often than needed in progress callbacks.

incrementPutStartStatistics

void incrementPutStartStatistics(long bytes)

At the start of a put/multipart upload operation, update the relevant counters.

Parameters:

bytes - bytes in the request.

incrementPutCompletedStatistics

void incrementPutCompletedStatistics(boolean success,
                                     long bytes)

At the end of a put/multipart upload operation, update the relevant counters and gauges.

Parameters:

success - did the operation succeed?

bytes - bytes in the request.

incrementPutProgressStatistics

void incrementPutProgressStatistics(String key,
                                    long bytes)

Callback for use in progress callbacks from put/multipart upload events. Increments those statistics which are expected to be updated during the ongoing upload operation.

Parameters:

key - key to file that is being written (for logging)

bytes - bytes successfully uploaded.

nonNullDurationTrackerFactory

org.apache.hadoop.fs.statistics.DurationTrackerFactory nonNullDurationTrackerFactory(org.apache.hadoop.fs.statistics.DurationTrackerFactory factory)

Given a possibly null duration tracker factory, return a non-null one for use in tracking durations -either that or the FS tracker itself.

Parameters:

factory - factory.

Returns:

a non-null factory.

deleteObjects

@Retries.RetryRaw
Map.Entry<Duration,software.amazon.awssdk.services.s3.model.DeleteObjectsResponse> deleteObjects(software.amazon.awssdk.services.s3.model.DeleteObjectsRequest deleteRequest)
                                                                                                            throws MultiObjectDeleteException,
                                                                                                                   software.amazon.awssdk.core.exception.SdkException,
                                                                                                                   IOException

Perform a bulk object delete operation against S3. Increments the OBJECT_DELETE_REQUESTS and write operation statistics

OBJECT_DELETE_OBJECTS is updated with the actual number of objects deleted in the request.

Retry policy: retry untranslated; delete considered idempotent. If the request is throttled, this is logged in the throttle statistics, with the counter set to the number of keys, rather than the number of invocations of the delete operation. This is because S3 considers each key as one mutating operation on the store when updating its load counters on a specific partition of an S3 bucket. If only the request was measured, this operation would under-report. A write capacity will be requested proportional to the number of keys preset in the request and will be re-requested during retries such that retries throttle better. If the request is throttled, the time spent is recorded in a duration IOStat named STORE_IO_RATE_LIMITED_DURATION.

Parameters:

deleteRequest - keys to delete on the s3-backend

Returns:

the AWS response

Throws:

MultiObjectDeleteException - one or more of the keys could not be deleted.

software.amazon.awssdk.core.exception.SdkException - amazon-layer failure.

IOException - IO problems.

deleteObject

@Retries.RetryRaw
Map.Entry<Duration,Optional<software.amazon.awssdk.services.s3.model.DeleteObjectResponse>> deleteObject(software.amazon.awssdk.services.s3.model.DeleteObjectRequest request)
                                                                                                                    throws software.amazon.awssdk.core.exception.SdkException

Delete an object. Increments the OBJECT_DELETE_REQUESTS statistics.

Retry policy: retry untranslated; delete considered idempotent. 404 errors other than bucket not found are swallowed; this can be raised by third party stores (GCS).

A write capacity of 1 ( as it is signle object delete) will be requested before the delete call and will be re-requested during retries such that retries throttle better. If the request is throttled, the time spent is recorded in a duration IOStat named STORE_IO_RATE_LIMITED_DURATION. If an exception is caught and swallowed, the response will be empty; otherwise it will be the response from the delete operation.

Parameters:

request - request to make

Returns:

the total duration and response.

Throws:

software.amazon.awssdk.core.exception.SdkException - problems working with S3

IllegalArgumentException - if the request was rejected due to a mistaken attempt to delete the root directory.

headObject

@Retries.RetryRaw
software.amazon.awssdk.services.s3.model.HeadObjectResponse headObject(String key,
                                                                                         ChangeTracker changeTracker,
                                                                                         Invoker changeInvoker,
                                                                                         S3AFileSystemOperations fsHandler,
                                                                                         String operation)
                                                                                  throws IOException

Performs a HEAD request on an S3 object to retrieve its metadata.

Parameters:

key - The S3 object key to perform the HEAD operation on

changeTracker - Tracks changes to the object's metadata across operations

changeInvoker - The invoker responsible for executing the HEAD request with retries

fsHandler - Handler for filesystem-level operations and configurations

operation - Description of the operation being performed for tracking purposes

Returns:

HeadObjectResponse containing the object's metadata

Throws:

IOException - If the HEAD request fails, object doesn't exist, or other I/O errors occur

getRangedS3Object

@Retries.RetryRaw
software.amazon.awssdk.core.ResponseInputStream<software.amazon.awssdk.services.s3.model.GetObjectResponse> getRangedS3Object(String key,
                                                                                                                                                long start,
                                                                                                                                                long end)
                                                                                                                                         throws IOException

Retrieves a specific byte range of an S3 object as a stream.

Parameters:

key - The S3 object key to retrieve

start - The starting byte position (inclusive) of the range to retrieve

end - The ending byte position (inclusive) of the range to retrieve

Returns:

A ResponseInputStream containing the requested byte range of the S3 object

Throws:

IOException - If the object cannot be retrieved other I/O errors occur

See Also:

For additional metadata about the retrieved object

uploadPart

@Retries.OnceRaw
software.amazon.awssdk.services.s3.model.UploadPartResponse uploadPart(software.amazon.awssdk.services.s3.model.UploadPartRequest request,
                                                                                        software.amazon.awssdk.core.sync.RequestBody body,
                                                                                        org.apache.hadoop.fs.statistics.DurationTrackerFactory durationTrackerFactory)
                                                                                 throws software.amazon.awssdk.awscore.exception.AwsServiceException,
                                                                                        UncheckedIOException

Upload part of a multi-partition file. Increments the write and put counters. Important: this call does not close any input stream in the body.

Retry Policy: none.

Parameters:

durationTrackerFactory - duration tracker factory for operation

request - the upload part request.

body - the request body.

Returns:

the result of the operation.

Throws:

software.amazon.awssdk.awscore.exception.AwsServiceException - on problems

UncheckedIOException - failure to instantiate the s3 client

putObject

@Retries.OnceRaw
UploadInfo putObject(software.amazon.awssdk.services.s3.model.PutObjectRequest putObjectRequest,
                                      File file,
                                      ProgressableProgressListener listener)
                               throws IOException

Start a transfer-manager managed async PUT of an object, incrementing the put requests and put bytes counters.

It does not update the other counters, as existing code does that as progress callbacks come in. Byte length is calculated from the file length, or, if there is no file, from the content length of the header.

Because the operation is async, any stream supplied in the request must reference data (files, buffers) which stay valid until the upload completes. Retry policy: N/A: the transfer manager is performing the upload. Auditing: must be inside an audit span.

Parameters:

putObjectRequest - the request

file - the file to be uploaded

listener - the progress listener for the request

Returns:

the upload initiated

Throws:

IOException - if transfer manager creation failed.

waitForUploadCompletion

@Retries.OnceTranslated
software.amazon.awssdk.transfer.s3.model.CompletedFileUpload waitForUploadCompletion(String key,
                                                                                                             UploadInfo uploadInfo)
                                                                                                      throws IOException

Wait for an upload to complete. If the upload (or its result collection) failed, this is where the failure is raised as an AWS exception. Calls incrementPutCompletedStatistics(boolean, long) to update the statistics.

Parameters:

key - destination key

uploadInfo - upload to wait for

Returns:

the upload result

Throws:

IOException - IO failure

CancellationException - if the wait() was cancelled

completeMultipartUpload

@Retries.OnceRaw
software.amazon.awssdk.services.s3.model.CompleteMultipartUploadResponse completeMultipartUpload(software.amazon.awssdk.services.s3.model.CompleteMultipartUploadRequest request)

Complete a multipart upload.

Parameters:

request - request

Returns:

the response

getDirectoryAllocator

org.apache.hadoop.fs.LocalDirAllocator getDirectoryAllocator()

Get the directory allocator.

Returns:

the directory allocator

createTemporaryFileForWriting

File createTemporaryFileForWriting(String pathStr,
                                   long size,
                                   org.apache.hadoop.conf.Configuration conf)
                            throws IOException

Demand create the directory allocator, then create a temporary file. This does not mark the file for deletion when a process exits. Pass in a file size of LocalDirAllocator.SIZE_UNKNOWN if the size is unknown. LocalDirAllocator.createTmpFileForWrite(String, long, Configuration).

Parameters:

pathStr - prefix for the temporary file

size - the size of the file that is going to be written

conf - the Configuration object

Returns:

a unique temporary file

Throws:

IOException - IO problems

inputStreamHasCapability

boolean inputStreamHasCapability(String capability)

Return the capabilities of input streams created through the store.

Parameters:

capability - string to query the stream support for.

Returns:

capabilities declared supported in streams.

hasCapability

default boolean hasCapability(String capability)

The StreamCapabilities is part of ObjectInputStreamFactory. To avoid confusion with any other streams which may be added here: always return false.

Specified by:

hasCapability in interface org.apache.hadoop.fs.StreamCapabilities

Parameters:

capability - string to query the stream support for.

Returns:

false, always.