org.apache.hadoop.fs.s3a.impl

Class DirMarkerTracker

java.lang.Object

org.apache.hadoop.fs.s3a.impl.DirMarkerTracker

public class DirMarkerTracker
extends Object

Tracks directory markers which have been reported in object listings. This is needed for auditing and cleanup, including during rename operations.

Designed to be used while scanning through the results of listObject calls, where are we assume the results come in alphanumeric sort order and parent entries before children.

This lets as assume that we can identify all leaf markers as those markers which were added to set of leaf markers and not subsequently removed as a child entries were discovered.

To avoid scanning datastructures excessively, the path of the parent directory of the last file added is cached. This allows for a quick bailout when many children of the same directory are returned in a listing.

Consult the directory_markers document for details on this feature, including terminology.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DirMarkerTracker.Marker This is a marker entry stored in the map and returned as markers are deleted.

Constructor Summary

Constructors

Constructor and Description
DirMarkerTracker(org.apache.hadoop.fs.Path basePath, boolean recordSurplusMarkers) Construct.

Method Summary

Modifier and Type	Method and Description
List<DirMarkerTracker.Marker>	fileFound(org.apache.hadoop.fs.Path path, String key, S3ALocatedFileStatus source) A file has been found.
org.apache.hadoop.fs.Path	getBasePath() Get the base path of the tracker.
int	getFilesFound()
org.apache.hadoop.fs.Path	getLastDirChecked()
Map<org.apache.hadoop.fs.Path,DirMarkerTracker.Marker>	getLeafMarkers() Get the map of leaf markers.
int	getMarkersFound()
int	getObjectsFound() How many objects were found.
int	getScanCount()
Map<org.apache.hadoop.fs.Path,DirMarkerTracker.Marker>	getSurplusMarkers() Get the map of surplus markers.
List<DirMarkerTracker.Marker>	markerFound(org.apache.hadoop.fs.Path path, String key, S3ALocatedFileStatus source) A marker has been found; this may or may not be a leaf.
List<org.apache.hadoop.fs.Path>	removeAllowedMarkers(DirectoryPolicy policy) Scan the surplus marker list and remove from it all where the directory policy says "keep".
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

DirMarkerTracker

public DirMarkerTracker(org.apache.hadoop.fs.Path basePath,
                        boolean recordSurplusMarkers)

Construct.

The base path is currently only used for information rather than validating paths supplied in other methods.

Parameters:

basePath - base path of track

recordSurplusMarkers - save surplus markers to a map?

Method Detail

getBasePath

public org.apache.hadoop.fs.Path getBasePath()

Get the base path of the tracker.

Returns:

the path

markerFound

public List<DirMarkerTracker.Marker> markerFound(org.apache.hadoop.fs.Path path,
                                                 String key,
                                                 S3ALocatedFileStatus source)

A marker has been found; this may or may not be a leaf.

Trigger a move of all markers above it into the surplus map.

Parameters:

path - marker path

key - object key

source - listing source

Returns:

the surplus markers found.

fileFound

public List<DirMarkerTracker.Marker> fileFound(org.apache.hadoop.fs.Path path,
                                               String key,
                                               S3ALocatedFileStatus source)

A file has been found. Trigger a move of all markers above it into the surplus map.

Parameters:

path - marker path

key - object key

source - listing source

Returns:

the surplus markers found.

getLeafMarkers

public Map<org.apache.hadoop.fs.Path,DirMarkerTracker.Marker> getLeafMarkers()

Get the map of leaf markers.

Returns:

all leaf markers.

getSurplusMarkers

public Map<org.apache.hadoop.fs.Path,DirMarkerTracker.Marker> getSurplusMarkers()

Get the map of surplus markers.

Empty if they were not being recorded.

Returns:

all surplus markers.

getLastDirChecked

public org.apache.hadoop.fs.Path getLastDirChecked()

getObjectsFound

public int getObjectsFound()

How many objects were found.

Returns:

count

getScanCount

public int getScanCount()

getFilesFound

public int getFilesFound()

getMarkersFound

public int getMarkersFound()

toString

public String toString()

Overrides:

toString in class Object

removeAllowedMarkers

public List<org.apache.hadoop.fs.Path> removeAllowedMarkers(DirectoryPolicy policy)

Scan the surplus marker list and remove from it all where the directory policy says "keep". This is useful when auditing

Parameters:

policy - policy to use when auditing markers for inclusion/exclusion.

Returns:

list of markers stripped