org.apache.hadoop.fs.s3a

Interface WriteOperations

All Superinterfaces:

org.apache.hadoop.fs.store.audit.AuditSpanSource, AutoCloseable, Closeable

All Known Implementing Classes:

WriteOperationHelper

public interface WriteOperations
extends org.apache.hadoop.fs.store.audit.AuditSpanSource, Closeable

Operations to update the store. This is effectively a private internal API for classes used as part of the S3A implementation. New extension points SHOULD use this interface -provided there is no plan to backport to previous versions. In those situations, use `WriteOperationHelper` directly.

Since:

Hadoop 3.3.0

Method Summary

Modifier and Type	Method and Description
void	abortMultipartCommit(String destKey, String uploadId) Abort a multipart commit operation.
void	abortMultipartUpload(com.amazonaws.services.s3.model.MultipartUpload upload) Abort a multipart commit operation.
void	abortMultipartUpload(String destKey, String uploadId, boolean shouldRetry, Invoker.Retried retrying) Abort a multipart upload operation.
int	abortMultipartUploadsUnderPath(String prefix) Abort multipart uploads under a path: limited to the first few hundred.
com.amazonaws.services.s3.model.CompleteMultipartUploadResult	commitUpload(String destKey, String uploadId, List<com.amazonaws.services.s3.model.PartETag> partETags, long length, BulkOperationState operationState) This completes a multipart upload to the destination key via finalizeMultipartUpload().
com.amazonaws.services.s3.model.CompleteMultipartUploadResult	completeMPUwithRetries(String destKey, String uploadId, List<com.amazonaws.services.s3.model.PartETag> partETags, long length, AtomicInteger errorCount) This completes a multipart upload to the destination key via finalizeMultipartUpload().
com.amazonaws.services.s3.model.PutObjectRequest	createPutObjectRequest(String dest, File sourceFile) Create a PutObjectRequest request to upload a file.
com.amazonaws.services.s3.model.PutObjectRequest	createPutObjectRequest(String destKey, InputStream inputStream, long length, Map<String,String> headers) Create a PutObjectRequest request against the specific key.
org.apache.hadoop.conf.Configuration	getConf() Get the configuration of this instance; essentially the owning filesystem configuration.
void	incrementWriteOperations() Increment the write operation counter of the filesystem.
BulkOperationState	initiateCommitOperation(org.apache.hadoop.fs.Path path) Initiate a commit operation through any metastore.
String	initiateMultiPartUpload(String destKey) Start the multipart upload process.
BulkOperationState	initiateOperation(org.apache.hadoop.fs.Path path, BulkOperationState.OperationType operationType) Initiate a commit operation through any metastore.
List<com.amazonaws.services.s3.model.MultipartUpload>	listMultipartUploads(String prefix) Abort multipart uploads under a path: limited to the first few hundred.
com.amazonaws.services.s3.model.ObjectMetadata	newObjectMetadata(long length) Create a new object metadata instance.
com.amazonaws.services.s3.model.SelectObjectContentRequest	newSelectRequest(org.apache.hadoop.fs.Path path) Create a S3 Select request for the destination path.
com.amazonaws.services.s3.model.UploadPartRequest	newUploadPartRequest(String destKey, String uploadId, int partNumber, int size, InputStream uploadStream, File sourceFile, Long offset) Create and initialize a part request of a multipart upload.
com.amazonaws.services.s3.model.PutObjectResult	putObject(com.amazonaws.services.s3.model.PutObjectRequest putObjectRequest) PUT an object directly (i.e.
<T> T	retry(String action, String path, boolean idempotent, org.apache.hadoop.util.functional.CallableRaisingIOE<T> operation) Execute a function with retry processing.
void	revertCommit(String destKey, BulkOperationState operationState) Revert a commit by deleting the file.
com.amazonaws.services.s3.model.SelectObjectContentResult	select(org.apache.hadoop.fs.Path source, com.amazonaws.services.s3.model.SelectObjectContentRequest request, String action) Execute an S3 Select operation.
com.amazonaws.services.s3.transfer.model.UploadResult	uploadObject(com.amazonaws.services.s3.model.PutObjectRequest putObjectRequest) PUT an object via the transfer manager.
com.amazonaws.services.s3.model.UploadPartResult	uploadPart(com.amazonaws.services.s3.model.UploadPartRequest request) Upload part of a multi-partition file.
void	writeFailed(Exception ex) Callback on a write failure.
void	writeSuccessful(long length) Callback on a successful write.

Methods inherited from interface org.apache.hadoop.fs.store.audit.AuditSpanSource

createSpan

Methods inherited from interface java.io.Closeable

close

Method Detail

retry

<T> T retry(String action,
            String path,
            boolean idempotent,
            org.apache.hadoop.util.functional.CallableRaisingIOE<T> operation)
     throws IOException

Execute a function with retry processing.

Type Parameters:

T - type of return value

Parameters:

action - action to execute (used in error messages)

path - path of work (used in error messages)

idempotent - does the operation have semantics which mean that it can be retried even if was already executed?

operation - operation to execute

Returns:

the result of the call

Throws:

IOException - any IOE raised, or translated exception

createPutObjectRequest

com.amazonaws.services.s3.model.PutObjectRequest createPutObjectRequest(String destKey,
                                                                        InputStream inputStream,
                                                                        long length,
                                                                        @Nullable
                                                                        Map<String,String> headers)

Create a PutObjectRequest request against the specific key.

Parameters:

destKey - destination key

inputStream - source data.

length - size, if known. Use -1 for not known

headers - optional map of custom headers.

Returns:

the request

createPutObjectRequest

com.amazonaws.services.s3.model.PutObjectRequest createPutObjectRequest(String dest,
                                                                        File sourceFile)

Create a PutObjectRequest request to upload a file.

Parameters:

dest - key to PUT to.

sourceFile - source file

Returns:

the request

writeSuccessful

void writeSuccessful(long length)

Callback on a successful write.

Parameters:

length - length of the write

writeFailed

void writeFailed(Exception ex)

Callback on a write failure.

Parameters:

ex - Any exception raised which triggered the failure.

newObjectMetadata

com.amazonaws.services.s3.model.ObjectMetadata newObjectMetadata(long length)

Create a new object metadata instance. Any standard metadata headers are added here, for example: encryption.

Parameters:

length - size, if known. Use -1 for not known

Returns:

a new metadata instance

initiateMultiPartUpload

@Retries.RetryTranslated
String initiateMultiPartUpload(String destKey)
                                                 throws IOException

Start the multipart upload process. Retry policy: retrying, translated.

Parameters:

destKey - destination of upload

Returns:

the upload result containing the ID

Throws:

IOException - IO problem

completeMPUwithRetries

@Retries.RetryTranslated
com.amazonaws.services.s3.model.CompleteMultipartUploadResult completeMPUwithRetries(String destKey,
                                                                                                              String uploadId,
                                                                                                              List<com.amazonaws.services.s3.model.PartETag> partETags,
                                                                                                              long length,
                                                                                                              AtomicInteger errorCount)
                                                                                                       throws IOException

This completes a multipart upload to the destination key via finalizeMultipartUpload(). Retry policy: retrying, translated. Retries increment the errorCount counter.

Parameters:

destKey - destination

uploadId - multipart operation Id

partETags - list of partial uploads

length - length of the upload

errorCount - a counter incremented by 1 on every error; for use in statistics

Returns:

the result of the operation.

Throws:

IOException - if problems arose which could not be retried, or the retry count was exceeded

abortMultipartUpload

@Retries.RetryTranslated
void abortMultipartUpload(String destKey,
                                                   String uploadId,
                                                   boolean shouldRetry,
                                                   Invoker.Retried retrying)
                                            throws IOException

Abort a multipart upload operation.

Parameters:

destKey - destination key of the upload

uploadId - multipart operation Id

shouldRetry - should failures trigger a retry?

retrying - callback invoked on every retry

Throws:

IOException - failure to abort

FileNotFoundException - if the abort ID is unknown

abortMultipartUpload

@Retries.RetryTranslated
void abortMultipartUpload(com.amazonaws.services.s3.model.MultipartUpload upload)
                                            throws IOException

Abort a multipart commit operation.

Parameters:

upload - upload to abort.

Throws:

IOException - on problems.

abortMultipartUploadsUnderPath

@Retries.RetryTranslated
int abortMultipartUploadsUnderPath(String prefix)
                                                     throws IOException

Abort multipart uploads under a path: limited to the first few hundred.

Parameters:

prefix - prefix for uploads to abort

Returns:

a count of aborts

Throws:

IOException - trouble; FileNotFoundExceptions are swallowed.

listMultipartUploads

List<com.amazonaws.services.s3.model.MultipartUpload> listMultipartUploads(String prefix)
                                                                    throws IOException

Abort multipart uploads under a path: limited to the first few hundred.

Parameters:

prefix - prefix for uploads to abort

Returns:

a count of aborts

Throws:

IOException - trouble; FileNotFoundExceptions are swallowed.

abortMultipartCommit

@Retries.RetryTranslated
void abortMultipartCommit(String destKey,
                                                   String uploadId)
                                            throws IOException

Abort a multipart commit operation.

Parameters:

destKey - destination key of ongoing operation

uploadId - multipart operation Id

Throws:

IOException - on problems.

FileNotFoundException - if the abort ID is unknown

newUploadPartRequest

com.amazonaws.services.s3.model.UploadPartRequest newUploadPartRequest(String destKey,
                                                                       String uploadId,
                                                                       int partNumber,
                                                                       int size,
                                                                       InputStream uploadStream,
                                                                       File sourceFile,
                                                                       Long offset)
                                                                throws IOException

Create and initialize a part request of a multipart upload. Exactly one of: uploadStream or sourceFile must be specified. A subset of the file may be posted, by providing the starting point in offset and a length of block in size equal to or less than the remaining bytes.

Parameters:

destKey - destination key of ongoing operation

uploadId - ID of ongoing upload

partNumber - current part number of the upload

size - amount of data

uploadStream - source of data to upload

sourceFile - optional source file.

offset - offset in file to start reading.

Returns:

the request.

Throws:

IllegalArgumentException - if the parameters are invalid

org.apache.hadoop.fs.PathIOException - if the part number is out of range.

IOException

putObject

@Retries.RetryTranslated
com.amazonaws.services.s3.model.PutObjectResult putObject(com.amazonaws.services.s3.model.PutObjectRequest putObjectRequest)
                                                                            throws IOException

PUT an object directly (i.e. not via the transfer manager). Byte length is calculated from the file length, or, if there is no file, from the content length of the header.

Parameters:

putObjectRequest - the request

Returns:

the upload initiated

Throws:

IOException - on problems

uploadObject

@Retries.RetryTranslated
com.amazonaws.services.s3.transfer.model.UploadResult uploadObject(com.amazonaws.services.s3.model.PutObjectRequest putObjectRequest)
                                                                                     throws IOException

PUT an object via the transfer manager.

Parameters:

putObjectRequest - the request

Returns:

the result of the operation

Throws:

IOException - on problems

revertCommit

@Retries.OnceTranslated
void revertCommit(String destKey,
                                          @Nullable
                                          BulkOperationState operationState)
                                   throws IOException

Revert a commit by deleting the file. Relies on retry code in filesystem

Parameters:

destKey - destination key

operationState - operational state for a bulk update

Throws:

IOException - on problems

commitUpload

@Retries.RetryTranslated
com.amazonaws.services.s3.model.CompleteMultipartUploadResult commitUpload(String destKey,
                                                                                                    String uploadId,
                                                                                                    List<com.amazonaws.services.s3.model.PartETag> partETags,
                                                                                                    long length,
                                                                                                    @Nullable
                                                                                                    BulkOperationState operationState)
                                                                                             throws IOException

This completes a multipart upload to the destination key via finalizeMultipartUpload(). Retry policy: retrying, translated. Retries increment the errorCount counter.

Parameters:

destKey - destination

uploadId - multipart operation Id

partETags - list of partial uploads

length - length of the upload

operationState - operational state for a bulk update

Returns:

the result of the operation.

Throws:

IOException - if problems arose which could not be retried, or the retry count was exceeded

initiateCommitOperation

BulkOperationState initiateCommitOperation(org.apache.hadoop.fs.Path path)
                                    throws IOException

Initiate a commit operation through any metastore.

Parameters:

path - path under which the writes will all take place.

Returns:

an possibly null operation state from the metastore.

Throws:

IOException - failure to instantiate.

initiateOperation

BulkOperationState initiateOperation(org.apache.hadoop.fs.Path path,
                                     BulkOperationState.OperationType operationType)
                              throws IOException

Initiate a commit operation through any metastore.

Parameters:

path - path under which the writes will all take place.

operationType - operation to initiate

Returns:

an possibly null operation state from the metastore.

Throws:

IOException - failure to instantiate.

uploadPart

@Retries.RetryTranslated
com.amazonaws.services.s3.model.UploadPartResult uploadPart(com.amazonaws.services.s3.model.UploadPartRequest request)
                                                                              throws IOException

Upload part of a multi-partition file.

Parameters:

request - request

Returns:

the result of the operation.

Throws:

IOException - on problems

getConf

org.apache.hadoop.conf.Configuration getConf()

Get the configuration of this instance; essentially the owning filesystem configuration.

Returns:

the configuration.

newSelectRequest

com.amazonaws.services.s3.model.SelectObjectContentRequest newSelectRequest(org.apache.hadoop.fs.Path path)

Create a S3 Select request for the destination path. This does not build the query.

Parameters:

path - pre-qualified path for query

Returns:

the request

select

@Retries.RetryTranslated
com.amazonaws.services.s3.model.SelectObjectContentResult select(org.apache.hadoop.fs.Path source,
                                                                                          com.amazonaws.services.s3.model.SelectObjectContentRequest request,
                                                                                          String action)
                                                                                   throws IOException

Execute an S3 Select operation. On a failure, the request is only logged at debug to avoid the select exception being printed.

Parameters:

source - source for selection

request - Select request to issue.

action - the action for use in exception creation

Returns:

response

Throws:

IOException - failure

incrementWriteOperations

void incrementWriteOperations()

Increment the write operation counter of the filesystem.