org.apache.hadoop.fs.s3a.s3guard

Class MetadataStoreListFilesIterator

java.lang.Object

org.apache.hadoop.fs.s3a.s3guard.MetadataStoreListFilesIterator

All Implemented Interfaces:

org.apache.hadoop.fs.RemoteIterator<S3AFileStatus>

@InterfaceAudience.Private
 @InterfaceStability.Evolving
public class MetadataStoreListFilesIterator
extends Object
implements org.apache.hadoop.fs.RemoteIterator<S3AFileStatus>

MetadataStoreListFilesIterator is a RemoteIterator that is similar to DescendantsIterator but does not return directories that have (or may have) children, and will also provide access to the set of tombstones to allow recently deleted S3 objects to be filtered out from a corresponding request. In other words, it returns tombstones and the same set of objects that should exist in S3: empty directories, and files, and not other directories whose existence is inferred therefrom. For example, assume the consistent store contains metadata representing this file system structure:

 /dir1
 |-- dir2
 |   |-- file1
 |   `-- file2
 `-- dir3
     |-- dir4
     |   `-- file3
     |-- dir5
     |   `-- file4
     `-- dir6
 

Consider this code sample:

 final PathMetadata dir1 = get(new Path("/dir1"));
 for (MetadataStoreListFilesIterator files =
     new MetadataStoreListFilesIterator(dir1); files.hasNext(); ) {
   final FileStatus status = files.next().getFileStatus();
   System.out.printf("%s %s%n", status.isDirectory() ? 'D' : 'F',
       status.getPath());
 }
 

The output is:

 F /dir1/dir2/file1
 F /dir1/dir2/file2
 F /dir1/dir3/dir4/file3
 F /dir1/dir3/dir5/file4
 D /dir1/dir3/dir6
 

Field Summary

Fields

Modifier and Type	Field and Description
static org.slf4j.Logger	LOG

Constructor Summary

Constructors

Constructor and Description
MetadataStoreListFilesIterator(MetadataStore ms, PathMetadata meta, boolean allowAuthoritative)

Method Summary

Modifier and Type	Method and Description
boolean	hasNext()
boolean	isRecursivelyAuthoritative()
Set<org.apache.hadoop.fs.Path>	listTombstones()
S3AFileStatus	next()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

Constructor Detail

MetadataStoreListFilesIterator

public MetadataStoreListFilesIterator(MetadataStore ms,
                                      PathMetadata meta,
                                      boolean allowAuthoritative)
                               throws IOException

Throws:

IOException

Method Detail

hasNext

public boolean hasNext()

Specified by:

hasNext in interface org.apache.hadoop.fs.RemoteIterator<S3AFileStatus>

next

public S3AFileStatus next()

Specified by:

next in interface org.apache.hadoop.fs.RemoteIterator<S3AFileStatus>

isRecursivelyAuthoritative

public boolean isRecursivelyAuthoritative()

listTombstones

public Set<org.apache.hadoop.fs.Path> listTombstones()