org.apache.hadoop.fs.s3a.commit

Class MagicCommitIntegration

java.lang.Object

org.apache.hadoop.fs.s3a.impl.AbstractStoreOperation

org.apache.hadoop.fs.s3a.commit.MagicCommitIntegration

public class MagicCommitIntegration
extends AbstractStoreOperation

Adds the code needed for S3A to support magic committers. It's pulled out to keep S3A FS class slightly less complex. This class can be instantiated even when magic commit is disabled; in this case:

1. isMagicCommitPath(Path) will always return false.
2. isUnderMagicPath(Path) will always return false.
3. createTracker(Path, String, PutTrackerStatistics) will always return an instance of PutTracker.

Important

: must not directly or indirectly import a class which uses any datatype in hadoop-mapreduce.

Constructor Summary

Constructors

Constructor and Description
MagicCommitIntegration(S3AFileSystem owner, boolean magicCommitEnabled) Instantiate.

Method Summary

Modifier and Type	Method and Description
PutTracker	createTracker(org.apache.hadoop.fs.Path path, String key, PutTrackerStatistics trackerStatistics) Given a path and a key to that same path, create a tracker for it.
boolean	isMagicCommitEnabled() Is magic commit enabled?
boolean	isMagicCommitPath(org.apache.hadoop.fs.Path path) Predicate: is a path a magic commit path?
boolean	isUnderMagicPath(org.apache.hadoop.fs.Path path) Is this path in/under a magic path...regardless of file type.
String	keyOfFinalDestination(List<String> elements, String key) Given an (elements, key) pair, return the key of the final destination of the PUT, that is: where the final path is expected to go?

Methods inherited from class org.apache.hadoop.fs.s3a.impl.AbstractStoreOperation

activateAuditSpan, getAuditSpan, getStoreContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MagicCommitIntegration

public MagicCommitIntegration(S3AFileSystem owner,
                              boolean magicCommitEnabled)

Instantiate.

Parameters:

owner - owner class

magicCommitEnabled - is magic commit enabled.

Method Detail

keyOfFinalDestination

public String keyOfFinalDestination(List<String> elements,
                                    String key)

Given an (elements, key) pair, return the key of the final destination of the PUT, that is: where the final path is expected to go?

Parameters:

elements - path split to elements

key - key

Returns:

key for final put. If this is not a magic commit, the same as the key in.

createTracker

public PutTracker createTracker(org.apache.hadoop.fs.Path path,
                                String key,
                                PutTrackerStatistics trackerStatistics)

Given a path and a key to that same path, create a tracker for it. This specific tracker will be chosen based on whether or not the path is a magic one. Auditing: the span used to invoke this method will be the one used to create the write operation helper for the commit tracker.

Parameters:

path - path of nominal write

key - key of path of nominal write

trackerStatistics - tracker statistics

Returns:

the tracker for this operation.

isMagicCommitEnabled

public boolean isMagicCommitEnabled()

Is magic commit enabled?

Returns:

true if magic commit is turned on.

isMagicCommitPath

public boolean isMagicCommitPath(org.apache.hadoop.fs.Path path)

Predicate: is a path a magic commit path?

Parameters:

path - path to examine

Returns:

true if the path is or is under a magic directory

isUnderMagicPath

public boolean isUnderMagicPath(org.apache.hadoop.fs.Path path)

Is this path in/under a magic path...regardless of file type. This is used to optimize create() operations.

Parameters:

path - path to check

Returns:

true if the path is one a magic file write expects.