org.apache.hadoop.fs.s3a

Class Listing

java.lang.Object

org.apache.hadoop.fs.s3a.impl.AbstractStoreOperation

org.apache.hadoop.fs.s3a.Listing

@InterfaceAudience.Private
public class Listing
extends AbstractStoreOperation

Place for the S3A listing classes; keeps all the small classes under control. Spans passed in are attached to the listing iterators returned, but are not closed at the end of the iteration. This is because the same span

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Listing.AcceptAllButSelfAndS3nDirs Accept all entries except the base path and those which map to S3N pseudo directory markers.

Constructor Summary

Constructors

Constructor and Description
Listing(ListingOperationCallbacks listingOperationCallbacks, StoreContext storeContext)

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.fs.s3a.Listing.FileStatusListingIterator	createFileStatusListingIterator(org.apache.hadoop.fs.Path listPath, S3ListRequest request, org.apache.hadoop.fs.PathFilter filter, org.apache.hadoop.fs.s3a.Listing.FileStatusAcceptor acceptor, org.apache.hadoop.fs.store.audit.AuditSpan span) Create a FileStatus iterator against a path, with a given list object request.
org.apache.hadoop.fs.s3a.Listing.FileStatusListingIterator	createFileStatusListingIterator(org.apache.hadoop.fs.Path listPath, S3ListRequest request, org.apache.hadoop.fs.PathFilter filter, org.apache.hadoop.fs.s3a.Listing.FileStatusAcceptor acceptor, org.apache.hadoop.fs.RemoteIterator<S3AFileStatus> providedStatus, org.apache.hadoop.fs.store.audit.AuditSpan span) Create a FileStatus iterator against a path, with a given list object request.
S3ListRequest	createListObjectsRequest(String key, String delimiter, org.apache.hadoop.fs.store.audit.AuditSpan span)
org.apache.hadoop.fs.RemoteIterator<S3ALocatedFileStatus>	createLocatedFileStatusIterator(org.apache.hadoop.fs.RemoteIterator<S3AFileStatus> statusIterator) Create a located status iterator over a file status iterator.
org.apache.hadoop.fs.RemoteIterator<S3ALocatedFileStatus>	createSingleStatusIterator(S3ALocatedFileStatus status) Create a remote iterator from a single status entry.
org.apache.commons.lang3.tuple.Triple<org.apache.hadoop.fs.RemoteIterator<S3AFileStatus>,DirListingMetadata,Boolean>	getFileStatusesAssumingNonEmptyDir(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.store.audit.AuditSpan span) Calculate list of file statuses assuming path to be a non-empty directory.
org.apache.hadoop.fs.RemoteIterator<S3ALocatedFileStatus>	getListFilesAssumingDir(org.apache.hadoop.fs.Path path, boolean recursive, org.apache.hadoop.fs.s3a.Listing.FileStatusAcceptor acceptor, boolean collectTombstones, boolean forceNonAuthoritativeMS, org.apache.hadoop.fs.store.audit.AuditSpan span) List files under a path assuming the path to be a directory.
org.apache.hadoop.fs.RemoteIterator<S3ALocatedFileStatus>	getLocatedFileStatusIteratorForDir(org.apache.hadoop.fs.Path dir, org.apache.hadoop.fs.PathFilter filter, org.apache.hadoop.fs.store.audit.AuditSpan span) Generate list located status for a directory.
static org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus>	toLocatedFileStatusIterator(org.apache.hadoop.fs.RemoteIterator<? extends org.apache.hadoop.fs.LocatedFileStatus> iterator)
static org.apache.hadoop.fs.RemoteIterator<S3AFileStatus>	toProvidedFileStatusIterator(S3AFileStatus[] fileStatuses) Create a FileStatus iterator against a provided list of file status.

Methods inherited from class org.apache.hadoop.fs.s3a.impl.AbstractStoreOperation

activateAuditSpan, getAuditSpan, getStoreContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Listing

public Listing(ListingOperationCallbacks listingOperationCallbacks,
               StoreContext storeContext)

Method Detail

toProvidedFileStatusIterator

public static org.apache.hadoop.fs.RemoteIterator<S3AFileStatus> toProvidedFileStatusIterator(S3AFileStatus[] fileStatuses)

Create a FileStatus iterator against a provided list of file status.

Parameters:

fileStatuses - array of file status.

Returns:

the file status iterator.

createFileStatusListingIterator

@Retries.RetryRaw
public org.apache.hadoop.fs.s3a.Listing.FileStatusListingIterator createFileStatusListingIterator(org.apache.hadoop.fs.Path listPath,
                                                                                                                    S3ListRequest request,
                                                                                                                    org.apache.hadoop.fs.PathFilter filter,
                                                                                                                    org.apache.hadoop.fs.s3a.Listing.FileStatusAcceptor acceptor,
                                                                                                                    org.apache.hadoop.fs.store.audit.AuditSpan span)
                                                                                                             throws IOException

Create a FileStatus iterator against a path, with a given list object request.

Parameters:

listPath - path of the listing

request - initial request to make

filter - the filter on which paths to accept

acceptor - the class/predicate to decide which entries to accept in the listing based on the full file status.

span - audit span for this iterator

Returns:

the iterator

Throws:

IOException - IO Problems

createFileStatusListingIterator

@Retries.RetryRaw
public org.apache.hadoop.fs.s3a.Listing.FileStatusListingIterator createFileStatusListingIterator(org.apache.hadoop.fs.Path listPath,
                                                                                                                    S3ListRequest request,
                                                                                                                    org.apache.hadoop.fs.PathFilter filter,
                                                                                                                    org.apache.hadoop.fs.s3a.Listing.FileStatusAcceptor acceptor,
                                                                                                                    org.apache.hadoop.fs.RemoteIterator<S3AFileStatus> providedStatus,
                                                                                                                    org.apache.hadoop.fs.store.audit.AuditSpan span)
                                                                                                             throws IOException

Create a FileStatus iterator against a path, with a given list object request.

Parameters:

listPath - path of the listing

request - initial request to make

filter - the filter on which paths to accept

acceptor - the class/predicate to decide which entries to accept in the listing based on the full file status.

providedStatus - the provided list of file status, which may contain items that are not listed from source.

span - audit span for this iterator

Returns:

the iterator

Throws:

IOException - IO Problems

createLocatedFileStatusIterator

public org.apache.hadoop.fs.RemoteIterator<S3ALocatedFileStatus> createLocatedFileStatusIterator(org.apache.hadoop.fs.RemoteIterator<S3AFileStatus> statusIterator)

Create a located status iterator over a file status iterator.

Parameters:

statusIterator - an iterator over the remote status entries

Returns:

a new remote iterator

createSingleStatusIterator

public org.apache.hadoop.fs.RemoteIterator<S3ALocatedFileStatus> createSingleStatusIterator(S3ALocatedFileStatus status)

Create a remote iterator from a single status entry.

Parameters:

status - status

Returns:

iterator.

getListFilesAssumingDir

public org.apache.hadoop.fs.RemoteIterator<S3ALocatedFileStatus> getListFilesAssumingDir(org.apache.hadoop.fs.Path path,
                                                                                         boolean recursive,
                                                                                         org.apache.hadoop.fs.s3a.Listing.FileStatusAcceptor acceptor,
                                                                                         boolean collectTombstones,
                                                                                         boolean forceNonAuthoritativeMS,
                                                                                         org.apache.hadoop.fs.store.audit.AuditSpan span)
                                                                                  throws IOException

List files under a path assuming the path to be a directory.

Parameters:

path - input path.

recursive - recursive listing?

acceptor - file status filter

collectTombstones - should tombstones be collected from S3Guard?

forceNonAuthoritativeMS - forces metadata store to act like non authoritative. This is useful when listFiles output is used by import tool.

span - audit span for this iterator

Returns:

an iterator over listing.

Throws:

IOException - any exception.

getLocatedFileStatusIteratorForDir

public org.apache.hadoop.fs.RemoteIterator<S3ALocatedFileStatus> getLocatedFileStatusIteratorForDir(org.apache.hadoop.fs.Path dir,
                                                                                                    org.apache.hadoop.fs.PathFilter filter,
                                                                                                    org.apache.hadoop.fs.store.audit.AuditSpan span)
                                                                                             throws IOException

Generate list located status for a directory. Also performing tombstone reconciliation for guarded directories.

Parameters:

dir - directory to check.

filter - a path filter.

span - audit span for this iterator

Returns:

an iterator that traverses statuses of the given dir.

Throws:

IOException - in case of failure.

getFileStatusesAssumingNonEmptyDir

public org.apache.commons.lang3.tuple.Triple<org.apache.hadoop.fs.RemoteIterator<S3AFileStatus>,DirListingMetadata,Boolean> getFileStatusesAssumingNonEmptyDir(org.apache.hadoop.fs.Path path,
                                                                                                                                                               org.apache.hadoop.fs.store.audit.AuditSpan span)
                                                                                                                                                        throws IOException

Calculate list of file statuses assuming path to be a non-empty directory.

Parameters:

path - input path.

span - audit span for this iterator

Returns:

Triple of file statuses, metaData, auth flag.

Throws:

IOException - Any IO problems.

createListObjectsRequest

public S3ListRequest createListObjectsRequest(String key,
                                              String delimiter,
                                              org.apache.hadoop.fs.store.audit.AuditSpan span)

toLocatedFileStatusIterator

public static org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> toLocatedFileStatusIterator(org.apache.hadoop.fs.RemoteIterator<? extends org.apache.hadoop.fs.LocatedFileStatus> iterator)