org.apache.hadoop.fs.s3a.impl

Interface ListingOperationCallbacks

All Known Implementing Classes:

S3AFileSystem.ListingOperationCallbacksImpl

public interface ListingOperationCallbacks

These are all the callbacks which Listing operations need, derived from the actual appropriate S3AFileSystem methods.

Method Summary

Modifier and Type	Method and Description
boolean	allowAuthoritative(org.apache.hadoop.fs.Path p) Is the path for this instance considered authoritative on the client, that is: will listing/status operations only be handled by the metastore, with no fallback to S3.
CompletableFuture<S3ListResult>	continueListObjectsAsync(S3ListRequest request, S3ListResult prevResult, org.apache.hadoop.fs.statistics.DurationTrackerFactory trackerFactory, org.apache.hadoop.fs.store.audit.AuditSpan span) List the next set of objects.
S3ListRequest	createListObjectsRequest(String key, String delimiter, org.apache.hadoop.fs.store.audit.AuditSpan span) Create a ListObjectsRequest request against this bucket, with the maximum keys returned in a query set in the FS config.
long	getDefaultBlockSize(org.apache.hadoop.fs.Path path) Return the number of bytes that large input files should be optimally be split into to minimize I/O time.
int	getMaxKeys() Get the maximum key count.
ITtlTimeProvider	getUpdatedTtlTimeProvider() Get the updated time provider for the current fs instance.
CompletableFuture<S3ListResult>	listObjectsAsync(S3ListRequest request, org.apache.hadoop.fs.statistics.DurationTrackerFactory trackerFactory, org.apache.hadoop.fs.store.audit.AuditSpan span) Initiate a listObjectsAsync operation, incrementing metrics in the process.
S3ALocatedFileStatus	toLocatedFileStatus(S3AFileStatus status) Build a S3ALocatedFileStatus from a FileStatus instance.

Method Detail

listObjectsAsync

@Retries.RetryRaw
CompletableFuture<S3ListResult> listObjectsAsync(S3ListRequest request,
                                                                   org.apache.hadoop.fs.statistics.DurationTrackerFactory trackerFactory,
                                                                   org.apache.hadoop.fs.store.audit.AuditSpan span)
                                                            throws IOException

Initiate a listObjectsAsync operation, incrementing metrics in the process. Retry policy: retry untranslated.

Parameters:

request - request to initiate

trackerFactory - tracker with statistics to update

span - audit span for this operation

Returns:

the results

Throws:

IOException - if the retry invocation raises one (it shouldn't).

continueListObjectsAsync

@Retries.RetryRaw
CompletableFuture<S3ListResult> continueListObjectsAsync(S3ListRequest request,
                                                                           S3ListResult prevResult,
                                                                           org.apache.hadoop.fs.statistics.DurationTrackerFactory trackerFactory,
                                                                           org.apache.hadoop.fs.store.audit.AuditSpan span)
                                                                    throws IOException

List the next set of objects. Retry policy: retry untranslated.

Parameters:

request - last list objects request to continue

prevResult - last paged result to continue from

trackerFactory - tracker with statistics to update

span - audit span for the IO

Returns:

the next result object

Throws:

IOException - none, just there for retryUntranslated.

toLocatedFileStatus

S3ALocatedFileStatus toLocatedFileStatus(S3AFileStatus status)
                                  throws IOException

Build a S3ALocatedFileStatus from a FileStatus instance.

Parameters:

status - file status

Returns:

a located status with block locations set up from this FS.

Throws:

IOException - IO Problems.

createListObjectsRequest

S3ListRequest createListObjectsRequest(String key,
                                       String delimiter,
                                       org.apache.hadoop.fs.store.audit.AuditSpan span)

Create a ListObjectsRequest request against this bucket, with the maximum keys returned in a query set in the FS config. The active span for the FS is handed the request to prepare it before this method returns. getMaxKeys().

Parameters:

key - key for request

delimiter - any delimiter

span - span within which the request takes place.

Returns:

the request

getDefaultBlockSize

long getDefaultBlockSize(org.apache.hadoop.fs.Path path)

Return the number of bytes that large input files should be optimally be split into to minimize I/O time. The given path will be used to locate the actual filesystem. The full path does not have to exist.

Parameters:

path - path of file

Returns:

the default block size for the path's filesystem

getMaxKeys

int getMaxKeys()

Get the maximum key count.

Returns:

a value, valid after initialization

getUpdatedTtlTimeProvider

ITtlTimeProvider getUpdatedTtlTimeProvider()

Get the updated time provider for the current fs instance.

Returns:

implementation of ITtlTimeProvider

allowAuthoritative

boolean allowAuthoritative(org.apache.hadoop.fs.Path p)

Is the path for this instance considered authoritative on the client, that is: will listing/status operations only be handled by the metastore, with no fallback to S3.

Parameters:

p - path

Returns:

true iff the path is authoritative on the client.