org.apache.hadoop.fs.s3a.impl

Interface OperationCallbacks

public interface OperationCallbacks

These are all the callbacks which the RenameOperation and DeleteOperation operations need, derived from the appropriate S3AFileSystem methods.

Method Summary

Modifier and Type	Method and Description
default long	abortMultipartUploadsUnderPrefix(String prefix) Abort multipart uploads under a path; paged.
software.amazon.awssdk.services.s3.model.CopyObjectResponse	copyFile(String srcKey, String destKey, S3ObjectAttributes srcAttributes, S3AReadOpContext readContext) Copy a single object in the bucket via a COPY operation.
S3ObjectAttributes	createObjectAttributes(org.apache.hadoop.fs.Path path, String eTag, String versionId, long len) Create the attributes of an object for subsequent use.
S3ObjectAttributes	createObjectAttributes(S3AFileStatus fileStatus) Create the attributes of an object for subsequent use.
S3AReadOpContext	createReadContext(org.apache.hadoop.fs.FileStatus fileStatus) Create the read context for reading from the referenced file, using FS state as well as the status.
void	deleteObjectAtPath(org.apache.hadoop.fs.Path path, String key, boolean isFile) Delete an object.
void	finishRename(org.apache.hadoop.fs.Path sourceRenamed, org.apache.hadoop.fs.Path destCreated) The rename has finished; perform any store cleanup operations such as creating/deleting directory markers.
org.apache.hadoop.fs.RemoteIterator<S3ALocatedFileStatus>	listFilesAndDirectoryMarkers(org.apache.hadoop.fs.Path path, S3AFileStatus status, boolean includeSelf) Recursive list of files and directory markers.
org.apache.hadoop.fs.RemoteIterator<S3AFileStatus>	listObjects(org.apache.hadoop.fs.Path path, String key) Create an iterator over objects in S3.
void	removeKeys(List<software.amazon.awssdk.services.s3.model.ObjectIdentifier> keysToDelete, boolean deleteFakeDir) Remove keys from the store.

Method Detail

createObjectAttributes

S3ObjectAttributes createObjectAttributes(org.apache.hadoop.fs.Path path,
                                          String eTag,
                                          String versionId,
                                          long len)

Create the attributes of an object for subsequent use.

Parameters:

path - path path of the request.

eTag - the eTag of the S3 object

versionId - S3 object version ID

len - length of the file

Returns:

attributes to use when building the query.

createObjectAttributes

S3ObjectAttributes createObjectAttributes(S3AFileStatus fileStatus)

Create the attributes of an object for subsequent use.

Parameters:

fileStatus - file status to build from.

Returns:

attributes to use when building the query.

createReadContext

S3AReadOpContext createReadContext(org.apache.hadoop.fs.FileStatus fileStatus)

Create the read context for reading from the referenced file, using FS state as well as the status.

Parameters:

fileStatus - file status.

Returns:

a context for read operations.

finishRename

void finishRename(org.apache.hadoop.fs.Path sourceRenamed,
                  org.apache.hadoop.fs.Path destCreated)
           throws IOException

The rename has finished; perform any store cleanup operations such as creating/deleting directory markers.

Parameters:

sourceRenamed - renamed source

destCreated - destination file created.

Throws:

IOException - failure

deleteObjectAtPath

@Retries.RetryTranslated
void deleteObjectAtPath(org.apache.hadoop.fs.Path path,
                                                 String key,
                                                 boolean isFile)
                                          throws IOException

Delete an object. This call does not create any mock parent entries. Retry policy: retry untranslated; delete considered idempotent.

Parameters:

path - path to delete

key - key of entry

isFile - is the path a file (used for instrumentation only)

Throws:

IOException - from invoker signature only -should not be raised.

listFilesAndDirectoryMarkers

@Retries.RetryTranslated
org.apache.hadoop.fs.RemoteIterator<S3ALocatedFileStatus> listFilesAndDirectoryMarkers(org.apache.hadoop.fs.Path path,
                                                                                                                S3AFileStatus status,
                                                                                                                boolean includeSelf)
                                                                                                         throws IOException

Recursive list of files and directory markers.

Parameters:

path - path to list from

status - optional status of path to list.

includeSelf - should the listing include this path if present?

Returns:

an iterator.

Throws:

IOException - failure

copyFile

@Retries.RetryTranslated
software.amazon.awssdk.services.s3.model.CopyObjectResponse copyFile(String srcKey,
                                                                                              String destKey,
                                                                                              S3ObjectAttributes srcAttributes,
                                                                                              S3AReadOpContext readContext)
                                                                                       throws IOException

Copy a single object in the bucket via a COPY operation. There's no update of metadata, directory markers, etc. Callers must implement.

Parameters:

srcKey - source object path

destKey - destination object path

srcAttributes - S3 attributes of the source object

readContext - the read context

Returns:

the result of the copy

Throws:

InterruptedIOException - the operation was interrupted

IOException - Other IO problems

removeKeys

@Retries.RetryRaw
void removeKeys(List<software.amazon.awssdk.services.s3.model.ObjectIdentifier> keysToDelete,
                                  boolean deleteFakeDir)
                           throws MultiObjectDeleteException,
                                  software.amazon.awssdk.awscore.exception.AwsServiceException,
                                  IOException

Remove keys from the store.

Parameters:

keysToDelete - collection of keys to delete on the s3-backend. if empty, no request is made of the object store.

deleteFakeDir - indicates whether this is for deleting fake dirs.

Throws:

org.apache.hadoop.fs.InvalidRequestException - if the request was rejected due to a mistaken attempt to delete the root directory.

MultiObjectDeleteException - one or more of the keys could not be deleted in a multiple object delete operation.

software.amazon.awssdk.awscore.exception.AwsServiceException - amazon-layer failure.

IOException - other IO Exception.

listObjects

@Retries.RetryTranslated
org.apache.hadoop.fs.RemoteIterator<S3AFileStatus> listObjects(org.apache.hadoop.fs.Path path,
                                                                                        String key)
                                                                                 throws IOException

Create an iterator over objects in S3. The listing includes the key itself, if found.

Parameters:

path - path of the listing.

key - object key

Returns:

iterator with the first listing completed.

Throws:

IOException - failure.

abortMultipartUploadsUnderPrefix

@Retries.RetryTranslated
default long abortMultipartUploadsUnderPrefix(String prefix)
                                                                throws IOException

Abort multipart uploads under a path; paged.

Parameters:

prefix - prefix for uploads to abort

Returns:

a count of aborts

Throws:

IOException - trouble; FileNotFoundExceptions are swallowed.