org.apache.hadoop.fs.s3a.impl

Class RenameOperation

java.lang.Object

org.apache.hadoop.fs.s3a.impl.AbstractStoreOperation

org.apache.hadoop.fs.s3a.impl.ExecutingStoreOperation<Long>

org.apache.hadoop.fs.s3a.impl.RenameOperation

All Implemented Interfaces:

org.apache.hadoop.util.functional.CallableRaisingIOE<Long>

public class RenameOperation
extends ExecutingStoreOperation<Long>

A parallelized rename operation which updates the metastore in the process, through whichever RenameTracker the store provides.

The parallel execution is in groups of size InternalConstants.RENAME_PARALLEL_LIMIT; it is only after one group completes that the next group is initiated.

Once enough files have been copied that they meet the InternalConstants.MAX_ENTRIES_TO_DELETE threshold, a delete is initiated. If it succeeds, the rename continues with the next group of files.

The RenameTracker has the task of keeping the metastore up to date as the rename proceeds.

Directory Markers which have child entries are never copied; only those which represent empty directories are copied in the rename. The DirMarkerTracker tracks which markers must be copied, and which can simply be deleted from the source. As a result: rename always purges all non-leaf directory markers from the copied tree. This is to ensure that even if a directory tree is copied from an authoritative path to a non-authoritative one there is never any contamination of the non-auth path with markers.

The rename operation implements the classic HDFS rename policy of rename(file, dir) renames the file under the directory.

There is no validation of input and output paths. Callers are required to themselves verify that destination is not under the source, above the source, the source itself, etc, etc.

Constructor Summary

Constructors

Constructor and Description
RenameOperation(StoreContext storeContext, org.apache.hadoop.fs.Path sourcePath, String sourceKey, S3AFileStatus sourceStatus, org.apache.hadoop.fs.Path destPath, String destKey, S3AFileStatus destStatus, OperationCallbacks callbacks, int pageSize) Initiate the rename.

Method Summary

Modifier and Type	Method and Description
Long	execute() Execute the operation.
protected CompletableFuture<org.apache.hadoop.fs.Path>	initiateCopy(S3ALocatedFileStatus source, String key, org.apache.hadoop.fs.Path childSourcePath, String newDestKey, org.apache.hadoop.fs.Path childDestPath) Initiate a copy operation in the executor.
protected void	recursiveDirectoryRename() Execute a full recursive rename.
protected org.apache.hadoop.fs.Path	renameFileToDest() The source is a file: rename it to the destination, which will be under the current destination path if that is a directory.

Methods inherited from class org.apache.hadoop.fs.s3a.impl.ExecutingStoreOperation

apply, executeOnlyOnce

Methods inherited from class org.apache.hadoop.fs.s3a.impl.AbstractStoreOperation

activateAuditSpan, getAuditSpan, getStoreContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RenameOperation

public RenameOperation(StoreContext storeContext,
                       org.apache.hadoop.fs.Path sourcePath,
                       String sourceKey,
                       S3AFileStatus sourceStatus,
                       org.apache.hadoop.fs.Path destPath,
                       String destKey,
                       S3AFileStatus destStatus,
                       OperationCallbacks callbacks,
                       int pageSize)

Initiate the rename.

Parameters:

storeContext - store context

sourcePath - source path

sourceKey - key of source

sourceStatus - pre-fetched source status

destPath - destination path.

destKey - destination key

destStatus - destination status.

callbacks - callback provider

pageSize - size of delete requests

Method Detail

execute

@Retries.RetryMixed
public Long execute()
                                 throws IOException

Description copied from class: ExecutingStoreOperation

Execute the operation. Subclasses MUST call ExecutingStoreOperation.executeOnlyOnce() so as to force the (atomic) re-entrancy check.

Specified by:

execute in class ExecutingStoreOperation<Long>

Returns:

the result.

Throws:

IOException - IO problem

renameFileToDest

protected org.apache.hadoop.fs.Path renameFileToDest()
                                              throws IOException

The source is a file: rename it to the destination, which will be under the current destination path if that is a directory.

Returns:

the path of the object created.

Throws:

IOException - failure

recursiveDirectoryRename

protected void recursiveDirectoryRename()
                                 throws IOException

Execute a full recursive rename. There is a special handling of directly markers here -only leaf markers are copied. This reduces incompatibility "regions" across versions.

Throws:

IOException - failure

initiateCopy

protected CompletableFuture<org.apache.hadoop.fs.Path> initiateCopy(S3ALocatedFileStatus source,
                                                                    String key,
                                                                    org.apache.hadoop.fs.Path childSourcePath,
                                                                    String newDestKey,
                                                                    org.apache.hadoop.fs.Path childDestPath)

Initiate a copy operation in the executor.

Parameters:

source - status of the source object.

key - source key

childSourcePath - source as a path.

newDestKey - destination key

childDestPath - destination path.

Returns:

the future.