org.apache.hadoop.fs.s3a.commit.staging

Class DirectoryStagingCommitter

java.lang.Object

org.apache.hadoop.mapreduce.OutputCommitter

org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter

org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter

org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter

org.apache.hadoop.fs.s3a.commit.staging.DirectoryStagingCommitter

All Implemented Interfaces:

org.apache.hadoop.fs.statistics.IOStatisticsSource

public class DirectoryStagingCommitter
extends StagingCommitter

This commits to a directory. The conflict policy is

• FAIL: fail the commit
• APPEND: add extra data to the destination.
• REPLACE: delete the destination directory in the job commit (i.e. after and only if all tasks have succeeded.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter

AbstractS3ACommitter.ActiveCommit, AbstractS3ACommitter.JobUUIDSource

Field Summary

Fields

Modifier and Type	Field and Description
static String	NAME Name: "directory".

Fields inherited from class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter

E_SELF_GENERATED_JOB_UUID, THREAD_PREFIX

Constructor Summary

Constructors

Constructor and Description
DirectoryStagingCommitter(org.apache.hadoop.fs.Path outputPath, org.apache.hadoop.mapreduce.TaskAttemptContext context)

Method Summary

Modifier and Type	Method and Description
String	getName() Get the name of this committer.
void	preCommitJob(CommitContext commitContext, AbstractS3ACommitter.ActiveCommit pending) Pre-commit actions for a job.
void	setupJob(org.apache.hadoop.mapreduce.JobContext context) Set up the job, including calling the same method on the wrapped committer.

Methods inherited from class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter

abortJobInternal, abortTask, cleanup, cleanupStagingDirs, commitTask, commitTaskInternal, createWrappedCommitter, deleteDestinationPaths, deleteStagingUploadsParentDirectory, deleteTaskWorkingPathQuietly, failDestinationExists, getBaseTaskAttemptPath, getCommittedTaskPath, getCommittedTaskPath, getConfictModeOption, getConflictResolutionMode, getFinalKey, getFinalPath, getJobAttemptFileSystem, getJobAttemptPath, getJobAttemptPath, getJobAttemptPath, getJobPath, getTaskAttemptPath, getTaskOutput, getTempTaskAttemptPath, initFileOutputCommitterOptions, listPendingUploads, listPendingUploadsToAbort, listPendingUploadsToCommit, needsTaskCommit, setupTask, toString, useUniqueFilenames

Methods inherited from class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter

abortJob, abortPendingUploads, abortPendingUploads, abortPendingUploadsInCleanup, buildJobUUID, cleanupJob, commitJob, commitJobInternal, commitPendingUploads, deleteTaskAttemptPathQuietly, getAuditSpanSource, getCommitOperations, getConf, getDestFS, getDestinationFS, getDestS3AFS, getIOStatistics, getJobContext, getOutputPath, getRole, getTaskAttemptFilesystem, getTaskAttemptPath, getUUID, getUUIDSource, getWorkPath, initiateJobOperation, initiateTaskOperation, initOutput, jobCompleted, maybeCreateSuccessMarker, maybeCreateSuccessMarkerFromCommits, maybeIgnore, maybeIgnore, precommitCheckPendingFiles, recoverTask, requiresDelayedCommitOutputInFileSystem, setConf, setDestFS, setOutputPath, setWorkPath, startOperation, updateCommonContext, warnOnActiveUploads

Methods inherited from class org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter

hasOutputPath

Methods inherited from class org.apache.hadoop.mapreduce.OutputCommitter

isCommitJobRepeatable, isRecoverySupported, isRecoverySupported

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

NAME

public static final String NAME

Name: "directory".

See Also:

Constant Field Values

Constructor Detail

DirectoryStagingCommitter

public DirectoryStagingCommitter(org.apache.hadoop.fs.Path outputPath,
                                 org.apache.hadoop.mapreduce.TaskAttemptContext context)
                          throws IOException

Throws:

IOException

Method Detail

getName

public String getName()

Description copied from class: AbstractS3ACommitter

Get the name of this committer.

Overrides:

getName in class StagingCommitter

Returns:

the committer name.

setupJob

public void setupJob(org.apache.hadoop.mapreduce.JobContext context)
              throws IOException

Description copied from class: StagingCommitter

Set up the job, including calling the same method on the wrapped committer.

Overrides:

setupJob in class StagingCommitter

Parameters:

context - job context

Throws:

IOException - IO failure.

preCommitJob

public void preCommitJob(CommitContext commitContext,
                         AbstractS3ACommitter.ActiveCommit pending)
                  throws IOException

Pre-commit actions for a job. Here: look at the conflict resolution mode and choose an action based on the current policy.

Overrides:

preCommitJob in class StagingCommitter

Parameters:

commitContext - commit context

pending - pending commits

Throws:

IOException - any failure