org.apache.hadoop.fs.s3a.s3guard

Class ProgressiveRenameTracker

java.lang.Object

org.apache.hadoop.fs.s3a.impl.AbstractStoreOperation

org.apache.hadoop.fs.s3a.s3guard.RenameTracker

org.apache.hadoop.fs.s3a.s3guard.ProgressiveRenameTracker

public class ProgressiveRenameTracker
extends RenameTracker

This rename tracker progressively updates the metadata store as it proceeds, during the parallelized copy operation.

Algorithm

1. As RenameTracker.fileCopied() callbacks are raised, the metastore is updated with the new file entry.
2. Including parent entries, as appropriate.
3. All directories which have been created are tracked locally, to avoid needing to read the store; this is a thread-safe structure.
4. The actual update is performed out of any synchronized block.
5. When deletes are executed, the store is also updated.
6. And at the completion of a successful rename, the source directory is also removed.

Field Summary

Fields inherited from class org.apache.hadoop.fs.s3a.s3guard.RenameTracker

LOG

Constructor Summary

Constructors

Constructor and Description
ProgressiveRenameTracker(StoreContext storeContext, MetadataStore metadataStore, org.apache.hadoop.fs.Path sourceRoot, org.apache.hadoop.fs.Path dest, BulkOperationState operationState)

Method Summary

Modifier and Type	Method and Description
void	completeRename() Complete the operation.
void	directoryMarkerCopied(org.apache.hadoop.fs.Path sourcePath, org.apache.hadoop.fs.Path destPath, boolean addAncestors) A directory marker has been added.
void	fileCopied(org.apache.hadoop.fs.Path sourcePath, S3ObjectAttributes sourceAttributes, S3ObjectAttributes destAttributes, org.apache.hadoop.fs.Path destPath, long blockSize, boolean addAncestors) When a file is copied, any ancestors are calculated and then the store is updated with the destination entries.
void	moveSourceDirectory() Top level directory move.
void	sourceObjectsDeleted(Collection<org.apache.hadoop.fs.Path> paths) As source objects are deleted, so is the list of entries.

Methods inherited from class org.apache.hadoop.fs.s3a.s3guard.RenameTracker

convertToIOException, deleteFailed, getDest, getMetadataStore, getOperationState, getOwner, getSourceRoot, noteRenameFinished, renameFailed, toString

Methods inherited from class org.apache.hadoop.fs.s3a.impl.AbstractStoreOperation

activateAuditSpan, getAuditSpan, getStoreContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

ProgressiveRenameTracker

public ProgressiveRenameTracker(StoreContext storeContext,
                                MetadataStore metadataStore,
                                org.apache.hadoop.fs.Path sourceRoot,
                                org.apache.hadoop.fs.Path dest,
                                BulkOperationState operationState)

Method Detail

fileCopied

public void fileCopied(org.apache.hadoop.fs.Path sourcePath,
                       S3ObjectAttributes sourceAttributes,
                       S3ObjectAttributes destAttributes,
                       org.apache.hadoop.fs.Path destPath,
                       long blockSize,
                       boolean addAncestors)
                throws IOException

When a file is copied, any ancestors are calculated and then the store is updated with the destination entries.

The source entries are added to the pathsToDelete list.

Specified by:

fileCopied in class RenameTracker

Parameters:

sourcePath - path of source

sourceAttributes - status of source.

destAttributes - destination attributes

destPath - destination path.

blockSize - block size.

addAncestors - should ancestors be added?

Throws:

IOException - failure

directoryMarkerCopied

public void directoryMarkerCopied(org.apache.hadoop.fs.Path sourcePath,
                                  org.apache.hadoop.fs.Path destPath,
                                  boolean addAncestors)
                           throws IOException

A directory marker has been added. Add the new entry and record the source path as another entry to delete.

Overrides:

directoryMarkerCopied in class RenameTracker

Parameters:

sourcePath - status of source.

destPath - destination path.

addAncestors - should ancestors be added?

Throws:

IOException - failure.

moveSourceDirectory

public void moveSourceDirectory()
                         throws IOException

Description copied from class: RenameTracker

Top level directory move. This is invoked after all child entries have been copied

Overrides:

moveSourceDirectory in class RenameTracker

Throws:

IOException - on failure

sourceObjectsDeleted

public void sourceObjectsDeleted(Collection<org.apache.hadoop.fs.Path> paths)
                          throws IOException

As source objects are deleted, so is the list of entries.

Overrides:

sourceObjectsDeleted in class RenameTracker

Parameters:

paths - path of objects deleted.

Throws:

IOException - failure.

completeRename

public void completeRename()
                    throws IOException

Description copied from class: RenameTracker

Complete the operation.

Overrides:

completeRename in class RenameTracker

Throws:

IOException - failure.