org.apache.hadoop.fs.s3a

Interface WriteOperations

All Superinterfaces:

org.apache.hadoop.fs.store.audit.AuditSpanSource, AutoCloseable, Closeable

All Known Implementing Classes:

WriteOperationHelper

public interface WriteOperations
extends org.apache.hadoop.fs.store.audit.AuditSpanSource, Closeable

Operations to update the store. This is effectively a private internal API for classes used as part of the S3A implementation. New extension points SHOULD use this interface -provided there is no plan to backport to previous versions. In those situations, use `WriteOperationHelper` directly.

Since:

Hadoop 3.3.0

Method Summary

Modifier and Type	Method and Description
void	abortMultipartCommit(String destKey, String uploadId) Abort a multipart commit operation.
void	abortMultipartUpload(software.amazon.awssdk.services.s3.model.MultipartUpload upload) Abort a multipart commit operation.
void	abortMultipartUpload(String destKey, String uploadId, boolean shouldRetry, Invoker.Retried retrying) Abort a multipart upload operation.
int	abortMultipartUploadsUnderPath(String prefix) Abort multipart uploads under a path: limited to the first few hundred.
software.amazon.awssdk.services.s3.model.CompleteMultipartUploadResponse	commitUpload(String destKey, String uploadId, List<software.amazon.awssdk.services.s3.model.CompletedPart> partETags, long length) This completes a multipart upload to the destination key via finalizeMultipartUpload().
software.amazon.awssdk.services.s3.model.CompleteMultipartUploadResponse	completeMPUwithRetries(String destKey, String uploadId, List<software.amazon.awssdk.services.s3.model.CompletedPart> partETags, long length, AtomicInteger errorCount, PutObjectOptions putOptions) This completes a multipart upload to the destination key via finalizeMultipartUpload().
software.amazon.awssdk.services.s3.model.PutObjectRequest	createPutObjectRequest(String destKey, long length, PutObjectOptions options, boolean isFile) Create a PutObjectRequest request against the specific key.
org.apache.hadoop.conf.Configuration	getConf() Get the configuration of this instance; essentially the owning filesystem configuration.
void	incrementWriteOperations() Increment the write operation counter of the filesystem.
String	initiateMultiPartUpload(String destKey, PutObjectOptions options) Start the multipart upload process.
List<software.amazon.awssdk.services.s3.model.MultipartUpload>	listMultipartUploads(String prefix) List in-progress multipart uploads under a path: limited to the first few hundred.
software.amazon.awssdk.services.s3.model.UploadPartRequest.Builder	newUploadPartRequestBuilder(String destKey, String uploadId, int partNumber, long size) Create and initialize a part request builder of a multipart upload.
software.amazon.awssdk.services.s3.model.PutObjectResponse	putObject(software.amazon.awssdk.services.s3.model.PutObjectRequest putObjectRequest, PutObjectOptions putOptions, S3ADataBlocks.BlockUploadData uploadData, boolean isFile, org.apache.hadoop.fs.statistics.DurationTrackerFactory durationTrackerFactory) PUT an object directly (i.e.
<T> T	retry(String action, String path, boolean idempotent, org.apache.hadoop.util.functional.CallableRaisingIOE<T> operation) Execute a function with retry processing.
void	revertCommit(String destKey) Revert a commit by deleting the file.
software.amazon.awssdk.services.s3.model.UploadPartResponse	uploadPart(software.amazon.awssdk.services.s3.model.UploadPartRequest request, software.amazon.awssdk.core.sync.RequestBody body, org.apache.hadoop.fs.statistics.DurationTrackerFactory durationTrackerFactory) Upload part of a multi-partition file.
void	writeFailed(Exception ex) Callback on a write failure.
void	writeSuccessful(long length) Callback on a successful write.

Methods inherited from interface org.apache.hadoop.fs.store.audit.AuditSpanSource

createSpan

Methods inherited from interface java.io.Closeable

close

Method Detail

retry

<T> T retry(String action,
            String path,
            boolean idempotent,
            org.apache.hadoop.util.functional.CallableRaisingIOE<T> operation)
     throws IOException

Execute a function with retry processing.

Type Parameters:

T - type of return value

Parameters:

action - action to execute (used in error messages)

path - path of work (used in error messages)

idempotent - does the operation have semantics which mean that it can be retried even if was already executed?

operation - operation to execute

Returns:

the result of the call

Throws:

IOException - any IOE raised, or translated exception

createPutObjectRequest

software.amazon.awssdk.services.s3.model.PutObjectRequest createPutObjectRequest(String destKey,
                                                                                 long length,
                                                                                 @Nullable
                                                                                 PutObjectOptions options,
                                                                                 boolean isFile)

Create a PutObjectRequest request against the specific key.

Parameters:

destKey - destination key

length - size, if known. Use -1 for not known

options - options for the request

isFile - is data to be uploaded a file

Returns:

the request

writeSuccessful

void writeSuccessful(long length)

Callback on a successful write.

Parameters:

length - length of the write

writeFailed

void writeFailed(Exception ex)

Callback on a write failure.

Parameters:

ex - Any exception raised which triggered the failure.

initiateMultiPartUpload

@Retries.RetryTranslated
String initiateMultiPartUpload(String destKey,
                                                        PutObjectOptions options)
                                                 throws IOException

Start the multipart upload process. Retry policy: retrying, translated.

Parameters:

destKey - destination of upload

options - options for the put request

Returns:

the upload result containing the ID

Throws:

IOException - IO problem

completeMPUwithRetries

@Retries.RetryTranslated
software.amazon.awssdk.services.s3.model.CompleteMultipartUploadResponse completeMPUwithRetries(String destKey,
                                                                                                                         String uploadId,
                                                                                                                         List<software.amazon.awssdk.services.s3.model.CompletedPart> partETags,
                                                                                                                         long length,
                                                                                                                         AtomicInteger errorCount,
                                                                                                                         PutObjectOptions putOptions)
                                                                                                                  throws IOException

This completes a multipart upload to the destination key via finalizeMultipartUpload(). Retry policy: retrying, translated. Retries increment the errorCount counter.

Parameters:

destKey - destination

uploadId - multipart operation Id

partETags - list of partial uploads

length - length of the upload

errorCount - a counter incremented by 1 on every error; for use in statistics

putOptions - put object options

Returns:

the result of the operation.

Throws:

IOException - if problems arose which could not be retried, or the retry count was exceeded

abortMultipartUpload

@Retries.RetryTranslated
void abortMultipartUpload(String destKey,
                                                   String uploadId,
                                                   boolean shouldRetry,
                                                   Invoker.Retried retrying)
                                            throws IOException

Abort a multipart upload operation.

Parameters:

destKey - destination key of the upload

uploadId - multipart operation Id

shouldRetry - should failures trigger a retry?

retrying - callback invoked on every retry

Throws:

IOException - failure to abort

FileNotFoundException - if the abort ID is unknown

abortMultipartUpload

@Retries.RetryTranslated
void abortMultipartUpload(software.amazon.awssdk.services.s3.model.MultipartUpload upload)
                                            throws IOException

Abort a multipart commit operation.

Parameters:

upload - upload to abort.

Throws:

IOException - on problems.

abortMultipartUploadsUnderPath

@Retries.RetryTranslated
int abortMultipartUploadsUnderPath(String prefix)
                                                     throws IOException

Abort multipart uploads under a path: limited to the first few hundred.

Parameters:

prefix - prefix for uploads to abort

Returns:

a count of aborts

Throws:

IOException - trouble; FileNotFoundExceptions are swallowed.

listMultipartUploads

List<software.amazon.awssdk.services.s3.model.MultipartUpload> listMultipartUploads(String prefix)
                                                                             throws IOException

List in-progress multipart uploads under a path: limited to the first few hundred.

Parameters:

prefix - prefix for uploads to list

Returns:

a list of in-progress multipart uploads

Throws:

IOException - on problems

abortMultipartCommit

@Retries.RetryTranslated
void abortMultipartCommit(String destKey,
                                                   String uploadId)
                                            throws IOException

Abort a multipart commit operation.

Parameters:

destKey - destination key of ongoing operation

uploadId - multipart operation Id

Throws:

IOException - on problems.

FileNotFoundException - if the abort ID is unknown

newUploadPartRequestBuilder

software.amazon.awssdk.services.s3.model.UploadPartRequest.Builder newUploadPartRequestBuilder(String destKey,
                                                                                               String uploadId,
                                                                                               int partNumber,
                                                                                               long size)
                                                                                        throws IOException

Create and initialize a part request builder of a multipart upload.

Parameters:

destKey - destination key of ongoing operation

uploadId - ID of ongoing upload

partNumber - current part number of the upload

size - amount of data

Returns:

the request builder.

Throws:

IllegalArgumentException - if the parameters are invalid

org.apache.hadoop.fs.PathIOException - if the part number is out of range.

IOException

putObject

@Retries.RetryTranslated
software.amazon.awssdk.services.s3.model.PutObjectResponse putObject(software.amazon.awssdk.services.s3.model.PutObjectRequest putObjectRequest,
                                                                                              PutObjectOptions putOptions,
                                                                                              S3ADataBlocks.BlockUploadData uploadData,
                                                                                              boolean isFile,
                                                                                              org.apache.hadoop.fs.statistics.DurationTrackerFactory durationTrackerFactory)
                                                                                       throws IOException

PUT an object directly (i.e. not via the transfer manager). Byte length is calculated from the file length, or, if there is no file, from the content length of the header.

Parameters:

putObjectRequest - the request

putOptions - put object options

durationTrackerFactory - factory for duration tracking

uploadData - data to be uploaded

isFile - is data to be uploaded a file

Returns:

the upload initiated

Throws:

IOException - on problems

revertCommit

@Retries.OnceTranslated
void revertCommit(String destKey)
                                   throws IOException

Revert a commit by deleting the file. No attempt is made to probe for/recreate a parent dir marker Relies on retry code in filesystem.

Parameters:

destKey - destination key

Throws:

IOException - on problems

commitUpload

@Retries.RetryTranslated
software.amazon.awssdk.services.s3.model.CompleteMultipartUploadResponse commitUpload(String destKey,
                                                                                                               String uploadId,
                                                                                                               List<software.amazon.awssdk.services.s3.model.CompletedPart> partETags,
                                                                                                               long length)
                                                                                                        throws IOException

This completes a multipart upload to the destination key via finalizeMultipartUpload(). Retry policy: retrying, translated. Retries increment the errorCount counter.

Parameters:

destKey - destination

uploadId - multipart operation Id

partETags - list of partial uploads

length - length of the upload

Returns:

the result of the operation.

Throws:

IOException - if problems arose which could not be retried, or the retry count was exceeded

uploadPart

@Retries.RetryTranslated
software.amazon.awssdk.services.s3.model.UploadPartResponse uploadPart(software.amazon.awssdk.services.s3.model.UploadPartRequest request,
                                                                                                software.amazon.awssdk.core.sync.RequestBody body,
                                                                                                org.apache.hadoop.fs.statistics.DurationTrackerFactory durationTrackerFactory)
                                                                                         throws IOException

Upload part of a multi-partition file.

Parameters:

request - the upload part request.

body - the request body.

durationTrackerFactory - factory for duration tracking

Returns:

the result of the operation.

Throws:

IOException - on problems

getConf

org.apache.hadoop.conf.Configuration getConf()

Get the configuration of this instance; essentially the owning filesystem configuration.

Returns:

the configuration.

incrementWriteOperations

void incrementWriteOperations()

Increment the write operation counter of the filesystem.