org.apache.hadoop.fs.s3a.commit.impl

Class CommitContext

java.lang.Object

org.apache.hadoop.fs.s3a.commit.impl.CommitContext

All Implemented Interfaces:

Closeable, AutoCloseable

public final class CommitContext
extends Object
implements Closeable

Commit context. It is used to manage the final commit sequence where files become visible. Once the commit operation has completed, it must be closed. It MUST NOT be reused. Audit integration: job and task attributes are added to the thread local context on create, removed on close(). JSON Serializers are created on demand, on a per thread basis. A WeakReferenceThreadMap is used here; a GC may lose the references, but they will recreated as needed.

Constructor Summary

Constructors

Constructor and Description
CommitContext(CommitOperations commitOperations, org.apache.hadoop.conf.Configuration conf, String jobId, int committerThreads, org.apache.hadoop.fs.statistics.IOStatisticsContext ioStatisticsContext) Create for testing.
CommitContext(CommitOperations commitOperations, org.apache.hadoop.mapreduce.JobContext jobContext, int committerThreads, org.apache.hadoop.fs.statistics.IOStatisticsContext ioStatisticsContext) Create.

Method Summary

Modifier and Type	Method and Description
void	abortMultipartCommit(String destKey, String uploadId) See CommitOperations.abortMultipartCommit(String, String)..
void	abortSingleCommit(SinglePendingCommit commit) See CommitOperations.abortSingleCommit(SinglePendingCommit).
void	close()
CommitOperations.MaybeIOE	commit(SinglePendingCommit commit, String origin) Commit a single pending commit; exceptions are caught and converted to an outcome.
void	commitOrFail(SinglePendingCommit commit) Commit the operation, throwing an exception on any failure.
org.apache.hadoop.conf.Configuration	getConf() Job configuration.
org.apache.hadoop.util.functional.TaskPool.Submitter	getInnerSubmitter() Return a submitter.
org.apache.hadoop.fs.statistics.IOStatisticsContext	getIOStatisticsContext() IOStatistics context of the created thread.
org.apache.hadoop.mapreduce.JobContext	getJobContext() Job Context.
String	getJobId() Get the job ID.
org.apache.hadoop.util.functional.TaskPool.Submitter	getOuterSubmitter() Return a submitter.
org.apache.hadoop.util.JsonSerialization<PendingSet>	getPendingSetSerializer() Get a serializer for .pendingset files.
org.apache.hadoop.util.JsonSerialization<SinglePendingCommit>	getSinglePendingFileSerializer() Get a serializer for .pending files.
boolean	isCollectIOStatistics() Collecting thread level IO statistics?
void	maybeResetIOStatisticsContext() Reset the IOStatistics context if statistics are being collected.
void	revertCommit(SinglePendingCommit commit) See CommitOperations.revertCommit(SinglePendingCommit).
void	switchToIOStatisticsContext() Switch to the context IOStatistics context, if needed.
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

CommitContext

public CommitContext(CommitOperations commitOperations,
                     org.apache.hadoop.mapreduce.JobContext jobContext,
                     int committerThreads,
                     org.apache.hadoop.fs.statistics.IOStatisticsContext ioStatisticsContext)

Create.

Parameters:

commitOperations - commit callbacks

jobContext - job context

committerThreads - number of commit threads

ioStatisticsContext - IOStatistics context of current thread

CommitContext

public CommitContext(CommitOperations commitOperations,
                     org.apache.hadoop.conf.Configuration conf,
                     String jobId,
                     int committerThreads,
                     org.apache.hadoop.fs.statistics.IOStatisticsContext ioStatisticsContext)

Create for testing. This has no job context; instead the values are set explicitly.

Parameters:

commitOperations - commit callbacks

conf - job conf

jobId - ID

committerThreads - number of commit threads

ioStatisticsContext - IOStatistics context of current thread

Method Detail

commitOrFail

public void commitOrFail(SinglePendingCommit commit)
                  throws IOException

Commit the operation, throwing an exception on any failure. See CommitOperations#commitOrFail(SinglePendingCommit).

Parameters:

commit - commit to execute

Throws:

IOException - on a failure

commit

public CommitOperations.MaybeIOE commit(SinglePendingCommit commit,
                                        String origin)

Commit a single pending commit; exceptions are caught and converted to an outcome. See CommitOperations.commit(SinglePendingCommit, String).

Parameters:

commit - entry to commit

origin - origin path/string for outcome text

Returns:

the outcome

abortSingleCommit

public void abortSingleCommit(SinglePendingCommit commit)
                       throws IOException

See CommitOperations.abortSingleCommit(SinglePendingCommit).

Parameters:

commit - pending commit to abort

Throws:

FileNotFoundException - if the abort ID is unknown

IOException - on any failure

revertCommit

public void revertCommit(SinglePendingCommit commit)
                  throws IOException

See CommitOperations.revertCommit(SinglePendingCommit).

Parameters:

commit - pending commit

Throws:

IOException - failure

abortMultipartCommit

public void abortMultipartCommit(String destKey,
                                 String uploadId)
                          throws IOException

See CommitOperations.abortMultipartCommit(String, String)..

Parameters:

destKey - destination key

uploadId - upload to cancel

Throws:

FileNotFoundException - if the abort ID is unknown

IOException - on any failure

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

toString

public String toString()

Overrides:

toString in class Object

getJobContext

public org.apache.hadoop.mapreduce.JobContext getJobContext()

Job Context.

Returns:

job context.

getOuterSubmitter

public org.apache.hadoop.util.functional.TaskPool.Submitter getOuterSubmitter()

Return a submitter. If created with 0 threads, this returns null so TaskPool knows to run it in the current thread.

Returns:

a submitter or null

getInnerSubmitter

public org.apache.hadoop.util.functional.TaskPool.Submitter getInnerSubmitter()

Return a submitter. As this pool is used less often, create it on demand. If created with 0 threads, this returns null so TaskPool knows to run it in the current thread.

Returns:

a submitter or null

getSinglePendingFileSerializer

public org.apache.hadoop.util.JsonSerialization<SinglePendingCommit> getSinglePendingFileSerializer()

Get a serializer for .pending files.

Returns:

a serializer.

getPendingSetSerializer

public org.apache.hadoop.util.JsonSerialization<PendingSet> getPendingSetSerializer()

Get a serializer for .pendingset files.

Returns:

a serializer.

getConf

public org.apache.hadoop.conf.Configuration getConf()

Job configuration.

Returns:

configuration (never null)

getJobId

public String getJobId()

Get the job ID.

Returns:

job ID.

isCollectIOStatistics

public boolean isCollectIOStatistics()

Collecting thread level IO statistics?

Returns:

true if thread level IO stats should be collected.

getIOStatisticsContext

public org.apache.hadoop.fs.statistics.IOStatisticsContext getIOStatisticsContext()

IOStatistics context of the created thread.

Returns:

the IOStatistics.

switchToIOStatisticsContext

public void switchToIOStatisticsContext()

Switch to the context IOStatistics context, if needed.

maybeResetIOStatisticsContext

public void maybeResetIOStatisticsContext()

Reset the IOStatistics context if statistics are being collected. Logs at info.