org.apache.hadoop.fs.s3a.impl

Interface OperationCallbacks

public interface OperationCallbacks

These are all the callbacks which the RenameOperation and DeleteOperation operations need, derived from the appropriate S3AFileSystem methods.

Method Summary

Modifier and Type	Method and Description
boolean	allowAuthoritative(org.apache.hadoop.fs.Path p) Is the path for this instance considered authoritative on the client, that is: will listing/status operations only be handled by the metastore, with no fallback to S3.
com.amazonaws.services.s3.transfer.model.CopyResult	copyFile(String srcKey, String destKey, S3ObjectAttributes srcAttributes, S3AReadOpContext readContext) Copy a single object in the bucket via a COPY operation.
S3ObjectAttributes	createObjectAttributes(org.apache.hadoop.fs.Path path, String eTag, String versionId, long len) Create the attributes of an object for subsequent use.
S3ObjectAttributes	createObjectAttributes(S3AFileStatus fileStatus) Create the attributes of an object for subsequent use.
S3AReadOpContext	createReadContext(org.apache.hadoop.fs.FileStatus fileStatus) Create the read context for reading from the referenced file, using FS state as well as the status.
void	deleteObjectAtPath(org.apache.hadoop.fs.Path path, String key, boolean isFile, BulkOperationState operationState) Delete an object, also updating the metastore.
void	finishRename(org.apache.hadoop.fs.Path sourceRenamed, org.apache.hadoop.fs.Path destCreated) The rename has finished; perform any store cleanup operations such as creating/deleting directory markers.
org.apache.hadoop.fs.RemoteIterator<S3ALocatedFileStatus>	listFilesAndDirectoryMarkers(org.apache.hadoop.fs.Path path, S3AFileStatus status, boolean collectTombstones, boolean includeSelf) Recursive list of files and directory markers.
org.apache.hadoop.fs.RemoteIterator<S3AFileStatus>	listObjects(org.apache.hadoop.fs.Path path, String key) Create an iterator over objects in S3 only; S3Guard is not involved.
com.amazonaws.services.s3.model.DeleteObjectsResult	removeKeys(List<com.amazonaws.services.s3.model.DeleteObjectsRequest.KeyVersion> keysToDelete, boolean deleteFakeDir, List<org.apache.hadoop.fs.Path> undeletedObjectsOnFailure, BulkOperationState operationState, boolean quiet) Remove keys from the store, updating the metastore on a partial delete represented as a MultiObjectDeleteException failure by deleting all those entries successfully deleted and then rethrowing the MultiObjectDeleteException.

Method Detail

createObjectAttributes

S3ObjectAttributes createObjectAttributes(org.apache.hadoop.fs.Path path,
                                          String eTag,
                                          String versionId,
                                          long len)

Create the attributes of an object for subsequent use.

Parameters:

path - path path of the request.

eTag - the eTag of the S3 object

versionId - S3 object version ID

len - length of the file

Returns:

attributes to use when building the query.

createObjectAttributes

S3ObjectAttributes createObjectAttributes(S3AFileStatus fileStatus)

Create the attributes of an object for subsequent use.

Parameters:

fileStatus - file status to build from.

Returns:

attributes to use when building the query.

createReadContext

S3AReadOpContext createReadContext(org.apache.hadoop.fs.FileStatus fileStatus)

Create the read context for reading from the referenced file, using FS state as well as the status.

Parameters:

fileStatus - file status.

Returns:

a context for read and select operations.

finishRename

void finishRename(org.apache.hadoop.fs.Path sourceRenamed,
                  org.apache.hadoop.fs.Path destCreated)
           throws IOException

The rename has finished; perform any store cleanup operations such as creating/deleting directory markers.

Parameters:

sourceRenamed - renamed source

destCreated - destination file created.

Throws:

IOException - failure

deleteObjectAtPath

@Retries.RetryTranslated
void deleteObjectAtPath(org.apache.hadoop.fs.Path path,
                                                 String key,
                                                 boolean isFile,
                                                 BulkOperationState operationState)
                                          throws IOException

Delete an object, also updating the metastore. This call does not create any mock parent entries. Retry policy: retry untranslated; delete considered idempotent.

Parameters:

path - path to delete

key - key of entry

isFile - is the path a file (used for instrumentation only)

operationState - (nullable) operational state for a bulk update

Throws:

com.amazonaws.AmazonClientException - problems working with S3

IOException - IO failure in the metastore

listFilesAndDirectoryMarkers

@Retries.RetryTranslated
org.apache.hadoop.fs.RemoteIterator<S3ALocatedFileStatus> listFilesAndDirectoryMarkers(org.apache.hadoop.fs.Path path,
                                                                                                                S3AFileStatus status,
                                                                                                                boolean collectTombstones,
                                                                                                                boolean includeSelf)
                                                                                                         throws IOException

Recursive list of files and directory markers.

Parameters:

path - path to list from

status - optional status of path to list.

collectTombstones - should tombstones be collected from S3Guard?

includeSelf - should the listing include this path if present?

Returns:

an iterator.

Throws:

IOException - failure

copyFile

@Retries.RetryTranslated
com.amazonaws.services.s3.transfer.model.CopyResult copyFile(String srcKey,
                                                                                      String destKey,
                                                                                      S3ObjectAttributes srcAttributes,
                                                                                      S3AReadOpContext readContext)
                                                                               throws IOException

Copy a single object in the bucket via a COPY operation. There's no update of metadata, directory markers, etc. Callers must implement.

Parameters:

srcKey - source object path

srcAttributes - S3 attributes of the source object

readContext - the read context

Returns:

the result of the copy

Throws:

InterruptedIOException - the operation was interrupted

IOException - Other IO problems

removeKeys

@Retries.RetryMixed
com.amazonaws.services.s3.model.DeleteObjectsResult removeKeys(List<com.amazonaws.services.s3.model.DeleteObjectsRequest.KeyVersion> keysToDelete,
                                                                                   boolean deleteFakeDir,
                                                                                   List<org.apache.hadoop.fs.Path> undeletedObjectsOnFailure,
                                                                                   BulkOperationState operationState,
                                                                                   boolean quiet)
                                                                            throws com.amazonaws.services.s3.model.MultiObjectDeleteException,
                                                                                   com.amazonaws.AmazonClientException,
                                                                                   IOException

Remove keys from the store, updating the metastore on a partial delete represented as a MultiObjectDeleteException failure by deleting all those entries successfully deleted and then rethrowing the MultiObjectDeleteException.

Parameters:

keysToDelete - collection of keys to delete on the s3-backend. if empty, no request is made of the object store.

deleteFakeDir - indicates whether this is for deleting fake dirs.

undeletedObjectsOnFailure - List which will be built up of all files that were not deleted. This happens even as an exception is raised.

operationState - bulk operation state

quiet - should a bulk query be quiet, or should its result list all deleted keys

Returns:

the deletion result if a multi object delete was invoked and it returned without a failure, else null.

Throws:

org.apache.hadoop.fs.InvalidRequestException - if the request was rejected due to a mistaken attempt to delete the root directory.

com.amazonaws.services.s3.model.MultiObjectDeleteException - one or more of the keys could not be deleted in a multiple object delete operation.

com.amazonaws.AmazonClientException - amazon-layer failure.

IOException - other IO Exception.

allowAuthoritative

boolean allowAuthoritative(org.apache.hadoop.fs.Path p)

Is the path for this instance considered authoritative on the client, that is: will listing/status operations only be handled by the metastore, with no fallback to S3.

Parameters:

p - path

Returns:

true iff the path is authoritative on the client.

listObjects

@Retries.RetryTranslated
org.apache.hadoop.fs.RemoteIterator<S3AFileStatus> listObjects(org.apache.hadoop.fs.Path path,
                                                                                        String key)
                                                                                 throws IOException

Create an iterator over objects in S3 only; S3Guard is not involved. The listing includes the key itself, if found.

Parameters:

path - path of the listing.

key - object key

Returns:

iterator with the first listing completed.

Throws:

IOException - failure.