org.apache.hadoop.fs.s3a.s3guard

Class DescendantsIterator

java.lang.Object

org.apache.hadoop.fs.s3a.s3guard.DescendantsIterator

All Implemented Interfaces:

org.apache.hadoop.fs.RemoteIterator<S3AFileStatus>

@InterfaceAudience.Private
 @InterfaceStability.Evolving
public class DescendantsIterator
extends Object
implements org.apache.hadoop.fs.RemoteIterator<S3AFileStatus>

DescendantsIterator is a RemoteIterator that implements pre-ordering breadth-first traversal (BFS) of a path and all of its descendants recursively. After visiting each path, that path's direct children are discovered by calling MetadataStore.listChildren(Path). Each iteration returns the next direct child, and if that child is a directory, also pushes it onto a queue to discover its children later. For example, assume the consistent store contains metadata representing this file system structure:

 /dir1
 |-- dir2
 |   |-- file1
 |   `-- file2
 `-- dir3
     |-- dir4
     |   `-- file3
     |-- dir5
     |   `-- file4
     `-- dir6
 

Consider this code sample:

 final PathMetadata dir1 = get(new Path("/dir1"));
 for (DescendantsIterator descendants = new DescendantsIterator(dir1);
     descendants.hasNext(); ) {
   final FileStatus status = descendants.next().getFileStatus();
   System.out.printf("%s %s%n", status.isDirectory() ? 'D' : 'F',
       status.getPath());
 }
 

The output is:

 D /dir1
 D /dir1/dir2
 D /dir1/dir3
 F /dir1/dir2/file1
 F /dir1/dir2/file2
 D /dir1/dir3/dir4
 D /dir1/dir3/dir5
 F /dir1/dir3/dir4/file3
 F /dir1/dir3/dir5/file4
 D /dir1/dir3/dir6
 

Constructor Summary

Constructors

Constructor and Description
DescendantsIterator(MetadataStore ms, PathMetadata meta) Creates a new DescendantsIterator.

Method Summary

Modifier and Type	Method and Description
boolean	hasNext()
S3AFileStatus	next()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DescendantsIterator

public DescendantsIterator(MetadataStore ms,
                           PathMetadata meta)
                    throws IOException

Creates a new DescendantsIterator.

Parameters:

ms - the associated MetadataStore

meta - base path for descendants iteration, which will be the first returned during iteration (except root). Null makes empty iterator.

Throws:

IOException - if errors happen during metadata store listing

Method Detail

hasNext

public boolean hasNext()
                throws IOException

Specified by:

hasNext in interface org.apache.hadoop.fs.RemoteIterator<S3AFileStatus>

Throws:

IOException

next

public S3AFileStatus next()
                   throws IOException

Specified by:

next in interface org.apache.hadoop.fs.RemoteIterator<S3AFileStatus>

Throws:

IOException