org.apache.hadoop.fs.s3a

Class WriteOperationHelper

java.lang.Object

org.apache.hadoop.fs.s3a.WriteOperationHelper

All Implemented Interfaces:

Closeable, AutoCloseable, WriteOperations, org.apache.hadoop.fs.store.audit.AuditSpanSource

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public class WriteOperationHelper
extends Object
implements WriteOperations

Helper for low-level operations against an S3 Bucket for writing data, creating and committing pending writes, and other S3-layer operations.

It hides direct access to the S3 API and is a location where the object operations can be evolved/enhanced.

Features

• Methods to create and submit requests to S3, so avoiding all direct interaction with the AWS APIs.
• Some extra preflight checks of arguments, so failing fast on errors.
• Callbacks to let the FS know of events in the output stream upload process.
• Other low-level access to S3 functions, for private use.
• Failure handling, including converting exceptions to IOEs.
• Integration with instrumentation and S3Guard.
• Evolution to add more low-level operations, such as S3 select.

This API is for internal use only. Span scoping: This helper is instantiated with span; it will be used before operations which query/update S3 History

 - A nested class in S3AFileSystem
 - Single shared instance created and reused.
 - [HADOOP-13786] A separate class, single instance in S3AFS
 - [HDFS-13934] Split into interface and implementation
 - [HADOOP-15711] Adds audit tracking; one instance per use.
 

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	WriteOperationHelper(S3AFileSystem owner, org.apache.hadoop.conf.Configuration conf, S3AStatisticsContext statisticsContext, org.apache.hadoop.fs.store.audit.AuditSpanSource auditSpanSource, org.apache.hadoop.fs.store.audit.AuditSpan auditSpan) Constructor.

Method Summary

Modifier and Type	Method and Description
void	abortMultipartCommit(String destKey, String uploadId) Abort a multipart commit operation.
void	abortMultipartUpload(com.amazonaws.services.s3.model.MultipartUpload upload) Abort a multipart commit operation.
void	abortMultipartUpload(String destKey, String uploadId, boolean shouldRetry, Invoker.Retried retrying) Abort a multipart upload operation.
int	abortMultipartUploadsUnderPath(String prefix) Abort multipart uploads under a path: limited to the first few hundred.
void	close() Deactivate the audit span.
com.amazonaws.services.s3.model.CompleteMultipartUploadResult	commitUpload(String destKey, String uploadId, List<com.amazonaws.services.s3.model.PartETag> partETags, long length, BulkOperationState operationState) This completes a multipart upload to the destination key via finalizeMultipartUpload().
com.amazonaws.services.s3.model.CompleteMultipartUploadResult	completeMPUwithRetries(String destKey, String uploadId, List<com.amazonaws.services.s3.model.PartETag> partETags, long length, AtomicInteger errorCount) This completes a multipart upload to the destination key via finalizeMultipartUpload().
com.amazonaws.services.s3.model.PutObjectRequest	createPutObjectRequest(String dest, File sourceFile) Create a PutObjectRequest request to upload a file.
com.amazonaws.services.s3.model.PutObjectRequest	createPutObjectRequest(String destKey, InputStream inputStream, long length, Map<String,String> headers) Create a PutObjectRequest request against the specific key.
org.apache.hadoop.fs.store.audit.AuditSpan	createSpan(String operation, String path1, String path2)
org.apache.hadoop.fs.store.audit.AuditSpan	getAuditSpan() Get the audit span this object was created with.
org.apache.hadoop.conf.Configuration	getConf() Get the configuration of this instance; essentially the owning filesystem configuration.
RequestFactory	getRequestFactory() Get the request factory which uses this store's audit span.
void	incrementWriteOperations() Increment the write operation counter of the filesystem.
BulkOperationState	initiateCommitOperation(org.apache.hadoop.fs.Path path) Initiate a commit operation through any metastore.
String	initiateMultiPartUpload(String destKey) Start the multipart upload process.
BulkOperationState	initiateOperation(org.apache.hadoop.fs.Path path, BulkOperationState.OperationType operationType) Initiate a commit operation through any metastore.
List<com.amazonaws.services.s3.model.MultipartUpload>	listMultipartUploads(String prefix) Abort multipart uploads under a path: limited to the first few hundred.
com.amazonaws.services.s3.model.ObjectMetadata	newObjectMetadata(long length) Create a new object metadata instance.
com.amazonaws.services.s3.model.SelectObjectContentRequest	newSelectRequest(org.apache.hadoop.fs.Path path) Create a S3 Select request for the destination path.
com.amazonaws.services.s3.model.UploadPartRequest	newUploadPartRequest(String destKey, String uploadId, int partNumber, int size, InputStream uploadStream, File sourceFile, Long offset) Create and initialize a part request of a multipart upload.
com.amazonaws.services.s3.model.PutObjectResult	putObject(com.amazonaws.services.s3.model.PutObjectRequest putObjectRequest) PUT an object directly (i.e.
<T> T	retry(String action, String path, boolean idempotent, org.apache.hadoop.util.functional.CallableRaisingIOE<T> operation) Execute a function with retry processing.
void	revertCommit(String destKey, BulkOperationState operationState) Revert a commit by deleting the file.
com.amazonaws.services.s3.model.SelectObjectContentResult	select(org.apache.hadoop.fs.Path source, com.amazonaws.services.s3.model.SelectObjectContentRequest request, String action) Execute an S3 Select operation.
String	toString() The toString method is intended to be used in logging/toString calls.
com.amazonaws.services.s3.transfer.model.UploadResult	uploadObject(com.amazonaws.services.s3.model.PutObjectRequest putObjectRequest) PUT an object via the transfer manager.
com.amazonaws.services.s3.model.UploadPartResult	uploadPart(com.amazonaws.services.s3.model.UploadPartRequest request) Upload part of a multi-partition file.
void	writeFailed(Exception ex) Callback on a write failure.
void	writeSuccessful(long length) Callback on a successful write.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

WriteOperationHelper

protected WriteOperationHelper(S3AFileSystem owner,
                               org.apache.hadoop.conf.Configuration conf,
                               S3AStatisticsContext statisticsContext,
                               org.apache.hadoop.fs.store.audit.AuditSpanSource auditSpanSource,
                               org.apache.hadoop.fs.store.audit.AuditSpan auditSpan)

Constructor.

Parameters:

owner - owner FS creating the helper

conf - Configuration object

statisticsContext - statistics context

auditSpanSource - source of spans

auditSpan - span to activate

Method Detail

retry

public <T> T retry(String action,
                   String path,
                   boolean idempotent,
                   org.apache.hadoop.util.functional.CallableRaisingIOE<T> operation)
            throws IOException

Execute a function with retry processing. Also activates the current span.

Specified by:

retry in interface WriteOperations

Type Parameters:

T - type of return value

Parameters:

action - action to execute (used in error messages)

path - path of work (used in error messages)

idempotent - does the operation have semantics which mean that it can be retried even if was already executed?

operation - operation to execute

Returns:

the result of the call

Throws:

IOException - any IOE raised, or translated exception

getAuditSpan

public org.apache.hadoop.fs.store.audit.AuditSpan getAuditSpan()

Get the audit span this object was created with.

Returns:

the audit span

createPutObjectRequest

@Retries.OnceRaw
public com.amazonaws.services.s3.model.PutObjectRequest createPutObjectRequest(String destKey,
                                                                                                InputStream inputStream,
                                                                                                long length,
                                                                                                Map<String,String> headers)

Create a PutObjectRequest request against the specific key.

Specified by:

createPutObjectRequest in interface WriteOperations

Parameters:

destKey - destination key

inputStream - source data.

length - size, if known. Use -1 for not known

headers - optional map of custom headers.

Returns:

the request

createPutObjectRequest

@Retries.OnceRaw
public com.amazonaws.services.s3.model.PutObjectRequest createPutObjectRequest(String dest,
                                                                                                File sourceFile)

Create a PutObjectRequest request to upload a file.

Specified by:

createPutObjectRequest in interface WriteOperations

Parameters:

dest - key to PUT to.

sourceFile - source file

Returns:

the request

writeSuccessful

public void writeSuccessful(long length)

Callback on a successful write.

Specified by:

writeSuccessful in interface WriteOperations

Parameters:

length - length of the write

writeFailed

public void writeFailed(Exception ex)

Callback on a write failure.

Specified by:

writeFailed in interface WriteOperations

Parameters:

ex - Any exception raised which triggered the failure.

newObjectMetadata

public com.amazonaws.services.s3.model.ObjectMetadata newObjectMetadata(long length)

Create a new object metadata instance. Any standard metadata headers are added here, for example: encryption.

Specified by:

newObjectMetadata in interface WriteOperations

Parameters:

length - size, if known. Use -1 for not known

Returns:

a new metadata instance

initiateMultiPartUpload

@Retries.RetryTranslated
public String initiateMultiPartUpload(String destKey)
                                                        throws IOException

Start the multipart upload process. Retry policy: retrying, translated.

Specified by:

initiateMultiPartUpload in interface WriteOperations

Parameters:

destKey - destination of upload

Returns:

the upload result containing the ID

Throws:

IOException - IO problem

completeMPUwithRetries

@Retries.RetryTranslated
public com.amazonaws.services.s3.model.CompleteMultipartUploadResult completeMPUwithRetries(String destKey,
                                                                                                                     String uploadId,
                                                                                                                     List<com.amazonaws.services.s3.model.PartETag> partETags,
                                                                                                                     long length,
                                                                                                                     AtomicInteger errorCount)
                                                                                                              throws IOException

This completes a multipart upload to the destination key via finalizeMultipartUpload(). Retry policy: retrying, translated. Retries increment the errorCount counter.

Specified by:

completeMPUwithRetries in interface WriteOperations

Parameters:

destKey - destination

uploadId - multipart operation Id

partETags - list of partial uploads

length - length of the upload

errorCount - a counter incremented by 1 on every error; for use in statistics

Returns:

the result of the operation.

Throws:

IOException - if problems arose which could not be retried, or the retry count was exceeded

abortMultipartUpload

@Retries.RetryTranslated
public void abortMultipartUpload(String destKey,
                                                          String uploadId,
                                                          boolean shouldRetry,
                                                          Invoker.Retried retrying)
                                                   throws IOException

Abort a multipart upload operation.

Specified by:

abortMultipartUpload in interface WriteOperations

Parameters:

destKey - destination key of the upload

uploadId - multipart operation Id

shouldRetry - should failures trigger a retry?

retrying - callback invoked on every retry

Throws:

IOException - failure to abort

FileNotFoundException - if the abort ID is unknown

abortMultipartUpload

@Retries.RetryTranslated
public void abortMultipartUpload(com.amazonaws.services.s3.model.MultipartUpload upload)
                                                   throws IOException

Abort a multipart commit operation.

Specified by:

abortMultipartUpload in interface WriteOperations

Parameters:

upload - upload to abort.

Throws:

IOException - on problems.

abortMultipartUploadsUnderPath

@Retries.RetryTranslated
public int abortMultipartUploadsUnderPath(String prefix)
                                                            throws IOException

Abort multipart uploads under a path: limited to the first few hundred.

Specified by:

abortMultipartUploadsUnderPath in interface WriteOperations

Parameters:

prefix - prefix for uploads to abort

Returns:

a count of aborts

Throws:

IOException - trouble; FileNotFoundExceptions are swallowed.

listMultipartUploads

@Retries.RetryTranslated
public List<com.amazonaws.services.s3.model.MultipartUpload> listMultipartUploads(String prefix)
                                                                                                    throws IOException

Description copied from interface: WriteOperations

Abort multipart uploads under a path: limited to the first few hundred.

Specified by:

listMultipartUploads in interface WriteOperations

Parameters:

prefix - prefix for uploads to abort

Returns:

a count of aborts

Throws:

IOException - trouble; FileNotFoundExceptions are swallowed.

abortMultipartCommit

@Retries.RetryTranslated
public void abortMultipartCommit(String destKey,
                                                          String uploadId)
                                                   throws IOException

Abort a multipart commit operation.

Specified by:

abortMultipartCommit in interface WriteOperations

Parameters:

destKey - destination key of ongoing operation

uploadId - multipart operation Id

Throws:

IOException - on problems.

FileNotFoundException - if the abort ID is unknown

newUploadPartRequest

@Retries.OnceTranslated
public com.amazonaws.services.s3.model.UploadPartRequest newUploadPartRequest(String destKey,
                                                                                                      String uploadId,
                                                                                                      int partNumber,
                                                                                                      int size,
                                                                                                      InputStream uploadStream,
                                                                                                      File sourceFile,
                                                                                                      Long offset)
                                                                                               throws IOException

Create and initialize a part request of a multipart upload. Exactly one of: uploadStream or sourceFile must be specified. A subset of the file may be posted, by providing the starting point in offset and a length of block in size equal to or less than the remaining bytes. The part number must be less than 10000. Retry policy is once-translated; to much effort

Specified by:

newUploadPartRequest in interface WriteOperations

Parameters:

destKey - destination key of ongoing operation

uploadId - ID of ongoing upload

partNumber - current part number of the upload

size - amount of data

uploadStream - source of data to upload

sourceFile - optional source file.

offset - offset in file to start reading.

Returns:

the request.

Throws:

IllegalArgumentException - if the parameters are invalid.

org.apache.hadoop.fs.PathIOException - if the part number is out of range.

IOException

toString

public String toString()

The toString method is intended to be used in logging/toString calls.

Overrides:

toString in class Object

Returns:

a string description.

putObject

@Retries.RetryTranslated
public com.amazonaws.services.s3.model.PutObjectResult putObject(com.amazonaws.services.s3.model.PutObjectRequest putObjectRequest)
                                                                                   throws IOException

PUT an object directly (i.e. not via the transfer manager). Byte length is calculated from the file length, or, if there is no file, from the content length of the header.

Specified by:

putObject in interface WriteOperations

Parameters:

putObjectRequest - the request

Returns:

the upload initiated

Throws:

IOException - on problems

uploadObject

@Retries.RetryTranslated
public com.amazonaws.services.s3.transfer.model.UploadResult uploadObject(com.amazonaws.services.s3.model.PutObjectRequest putObjectRequest)
                                                                                            throws IOException

PUT an object via the transfer manager.

Specified by:

uploadObject in interface WriteOperations

Parameters:

putObjectRequest - the request

Returns:

the result of the operation

Throws:

IOException - on problems

revertCommit

@Retries.OnceTranslated
public void revertCommit(String destKey,
                                                 @Nullable
                                                 BulkOperationState operationState)
                                          throws IOException

Revert a commit by deleting the file. Relies on retry code in filesystem

Specified by:

revertCommit in interface WriteOperations

Parameters:

destKey - destination key

operationState - operational state for a bulk update

Throws:

IOException - on problems

commitUpload

@Retries.RetryTranslated
public com.amazonaws.services.s3.model.CompleteMultipartUploadResult commitUpload(String destKey,
                                                                                                           String uploadId,
                                                                                                           List<com.amazonaws.services.s3.model.PartETag> partETags,
                                                                                                           long length,
                                                                                                           @Nullable
                                                                                                           BulkOperationState operationState)
                                                                                                    throws IOException

This completes a multipart upload to the destination key via finalizeMultipartUpload(). Retry policy: retrying, translated. Retries increment the errorCount counter.

Specified by:

commitUpload in interface WriteOperations

Parameters:

destKey - destination

uploadId - multipart operation Id

partETags - list of partial uploads

length - length of the upload

operationState - operational state for a bulk update

Returns:

the result of the operation.

Throws:

IOException - if problems arose which could not be retried, or the retry count was exceeded

initiateCommitOperation

public BulkOperationState initiateCommitOperation(org.apache.hadoop.fs.Path path)
                                           throws IOException

Initiate a commit operation through any metastore.

Specified by:

initiateCommitOperation in interface WriteOperations

Parameters:

path - path under which the writes will all take place.

Returns:

an possibly null operation state from the metastore.

Throws:

IOException - failure to instantiate.

initiateOperation

public BulkOperationState initiateOperation(org.apache.hadoop.fs.Path path,
                                            BulkOperationState.OperationType operationType)
                                     throws IOException

Initiate a commit operation through any metastore.

Specified by:

initiateOperation in interface WriteOperations

Parameters:

path - path under which the writes will all take place.

operationType - operation to initiate

Returns:

an possibly null operation state from the metastore.

Throws:

IOException - failure to instantiate.

uploadPart

@Retries.RetryTranslated
public com.amazonaws.services.s3.model.UploadPartResult uploadPart(com.amazonaws.services.s3.model.UploadPartRequest request)
                                                                                     throws IOException

Upload part of a multi-partition file.

Specified by:

uploadPart in interface WriteOperations

Parameters:

request - request

Returns:

the result of the operation.

Throws:

IOException - on problems

getConf

public org.apache.hadoop.conf.Configuration getConf()

Get the configuration of this instance; essentially the owning filesystem configuration.

Specified by:

getConf in interface WriteOperations

Returns:

the configuration.

newSelectRequest

public com.amazonaws.services.s3.model.SelectObjectContentRequest newSelectRequest(org.apache.hadoop.fs.Path path)

Create a S3 Select request for the destination path. This does not build the query.

Specified by:

newSelectRequest in interface WriteOperations

Parameters:

path - pre-qualified path for query

Returns:

the request

select

@Retries.RetryTranslated
public com.amazonaws.services.s3.model.SelectObjectContentResult select(org.apache.hadoop.fs.Path source,
                                                                                                 com.amazonaws.services.s3.model.SelectObjectContentRequest request,
                                                                                                 String action)
                                                                                          throws IOException

Execute an S3 Select operation. On a failure, the request is only logged at debug to avoid the select exception being printed.

Specified by:

select in interface WriteOperations

Parameters:

source - source for selection

request - Select request to issue.

action - the action for use in exception creation

Returns:

response

Throws:

IOException - failure

createSpan

public org.apache.hadoop.fs.store.audit.AuditSpan createSpan(String operation,
                                                             @Nullable
                                                             String path1,
                                                             @Nullable
                                                             String path2)
                                                      throws IOException

Specified by:

createSpan in interface org.apache.hadoop.fs.store.audit.AuditSpanSource

Throws:

IOException

incrementWriteOperations

public void incrementWriteOperations()

Description copied from interface: WriteOperations

Increment the write operation counter of the filesystem.

Specified by:

incrementWriteOperations in interface WriteOperations

close

public void close()
           throws IOException

Deactivate the audit span.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

getRequestFactory

public RequestFactory getRequestFactory()

Get the request factory which uses this store's audit span.

Returns:

the request factory.