org.apache.hadoop.fs.s3a.commit.staging

Class StagingCommitter

java.lang.Object

org.apache.hadoop.mapreduce.OutputCommitter

org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter

org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter

org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter

All Implemented Interfaces:

org.apache.hadoop.fs.statistics.IOStatisticsSource

Direct Known Subclasses:

DirectoryStagingCommitter, PartitionedStagingCommitter

public class StagingCommitter
extends AbstractS3ACommitter

Committer based on the contributed work of the Netflix multipart committers.

1. The working directory of each task is actually under a temporary path in the local filesystem; jobs write directly into it.
2. Task Commit: list all files under the task working dir, upload each of them but do not commit the final operation. Persist the information for each pending commit into the cluster for enumeration and commit by the job committer.
3. Task Abort: recursive delete of task working dir.
4. Job Commit: list all pending PUTs to commit; commit them.
5. Job Abort: list all pending PUTs to commit; abort them. Delete all task attempt directories.

This is the base class of the Partitioned and Directory committers. It does not do any conflict resolution, and is made non-abstract primarily for test purposes. It is not expected to be used in production.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter

AbstractS3ACommitter.ActiveCommit, AbstractS3ACommitter.JobUUIDSource

Field Summary

Fields

Modifier and Type	Field and Description
static String	NAME Name: "staging".

Fields inherited from class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter

E_SELF_GENERATED_JOB_UUID, THREAD_PREFIX

Constructor Summary

Constructors

Constructor and Description
StagingCommitter(org.apache.hadoop.fs.Path outputPath, org.apache.hadoop.mapreduce.TaskAttemptContext context) Committer for a single task attempt.

Method Summary

Modifier and Type	Method and Description
protected void	abortJobInternal(CommitContext commitContext, boolean suppressExceptions) The internal job abort operation; can be overridden in tests.
void	abortTask(org.apache.hadoop.mapreduce.TaskAttemptContext context) Abort the task.
protected void	cleanup(CommitContext commitContext, boolean suppressExceptions) Staging committer cleanup includes calling wrapped committer's cleanup method, and removing staging uploads path and all destination paths in the final filesystem.
void	cleanupStagingDirs() Clean up any staging directories.
void	commitTask(org.apache.hadoop.mapreduce.TaskAttemptContext context)
protected int	commitTaskInternal(org.apache.hadoop.mapreduce.TaskAttemptContext context, List<? extends org.apache.hadoop.fs.FileStatus> taskOutput, CommitContext commitContext) Commit the task by uploading all created files and then writing a pending entry for them.
protected org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter	createWrappedCommitter(org.apache.hadoop.mapreduce.JobContext context, org.apache.hadoop.conf.Configuration conf) Create the wrapped committer.
protected void	deleteDestinationPaths(org.apache.hadoop.mapreduce.JobContext context) Delete the working paths of a job.
protected void	deleteStagingUploadsParentDirectory(org.apache.hadoop.mapreduce.JobContext context) Delete the multipart upload staging directory.
protected void	deleteTaskWorkingPathQuietly(org.apache.hadoop.mapreduce.JobContext context) Delete the working path of a task; no-op if there is none, that is: this is a job.
protected org.apache.hadoop.fs.PathExistsException	failDestinationExists(org.apache.hadoop.fs.Path path, String description) Generate a PathExistsException because the destination exists.
org.apache.hadoop.fs.Path	getBaseTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context) Return the local work path as the destination for writing work.
protected org.apache.hadoop.fs.Path	getCommittedTaskPath(int appAttemptId, org.apache.hadoop.mapreduce.TaskAttemptContext context) Compute the path where the output of a committed task is stored until the entire job is committed for a specific application attempt.
org.apache.hadoop.fs.Path	getCommittedTaskPath(org.apache.hadoop.mapreduce.TaskAttemptContext context) Compute the path where the output of a committed task is stored until the entire job is committed.
static String	getConfictModeOption(org.apache.hadoop.mapreduce.JobContext context, org.apache.hadoop.conf.Configuration fsConf, String defVal) Get the conflict mode option string.
ConflictResolution	getConflictResolutionMode(org.apache.hadoop.mapreduce.JobContext context, org.apache.hadoop.conf.Configuration fsConf) Returns the ConflictResolution mode for this commit.
protected String	getFinalKey(String relative, org.apache.hadoop.mapreduce.JobContext context) Returns the final S3 key for a relative path.
protected org.apache.hadoop.fs.Path	getFinalPath(String relative, org.apache.hadoop.mapreduce.JobContext context) Returns the final S3 location for a relative path as a Hadoop Path.
org.apache.hadoop.fs.FileSystem	getJobAttemptFileSystem(org.apache.hadoop.mapreduce.JobContext context) Get the filesystem for the job attempt.
protected org.apache.hadoop.fs.Path	getJobAttemptPath(int appAttemptId) Compute the path where the output of a given job attempt will be placed.
org.apache.hadoop.fs.Path	getJobAttemptPath(org.apache.hadoop.mapreduce.JobContext context) For a job attempt path, the staging committer returns that of the wrapped committer.
static org.apache.hadoop.fs.Path	getJobAttemptPath(org.apache.hadoop.mapreduce.JobContext context, org.apache.hadoop.fs.Path out) Compute the path where the output of a given job attempt will be placed.
protected org.apache.hadoop.fs.Path	getJobPath() Compute the path under which all job attempts will be placed.
String	getName() Get the name of this committer.
static org.apache.hadoop.fs.Path	getTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context, org.apache.hadoop.fs.Path out) Compute the path where the output of a task attempt is stored until that task is committed.
protected List<org.apache.hadoop.fs.LocatedFileStatus>	getTaskOutput(org.apache.hadoop.mapreduce.TaskAttemptContext context) Lists the output of a task under the task attempt path.
org.apache.hadoop.fs.Path	getTempTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context) Get a temporary directory for data.
protected void	initFileOutputCommitterOptions(org.apache.hadoop.mapreduce.JobContext context) Init the context config with everything needed for the file output committer.
protected AbstractS3ACommitter.ActiveCommit	listPendingUploads(CommitContext commitContext, boolean suppressExceptions) Get the list of pending uploads for this job attempt.
protected AbstractS3ACommitter.ActiveCommit	listPendingUploadsToAbort(CommitContext commitContext) Get the list of pending uploads for this job attempt, swallowing exceptions.
protected AbstractS3ACommitter.ActiveCommit	listPendingUploadsToCommit(CommitContext commitContext) Get the list of pending uploads for this job attempt.
boolean	needsTaskCommit(org.apache.hadoop.mapreduce.TaskAttemptContext context)
void	preCommitJob(CommitContext commitContext, AbstractS3ACommitter.ActiveCommit pending) Pre-commit actions for a job.
void	setupJob(org.apache.hadoop.mapreduce.JobContext context) Set up the job, including calling the same method on the wrapped committer.
void	setupTask(org.apache.hadoop.mapreduce.TaskAttemptContext context) Task setup.
String	toString()
Boolean	useUniqueFilenames() Is this committer using unique filenames?

Methods inherited from class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter

abortJob, abortPendingUploads, abortPendingUploads, abortPendingUploadsInCleanup, buildJobUUID, cleanupJob, commitJob, commitJobInternal, commitPendingUploads, deleteTaskAttemptPathQuietly, getAuditSpanSource, getCommitOperations, getConf, getDestFS, getDestinationFS, getDestS3AFS, getIOStatistics, getJobContext, getOutputPath, getRole, getTaskAttemptFilesystem, getTaskAttemptPath, getUUID, getUUIDSource, getWorkPath, initiateJobOperation, initiateTaskOperation, initOutput, jobCompleted, maybeCreateSuccessMarker, maybeCreateSuccessMarkerFromCommits, maybeIgnore, maybeIgnore, precommitCheckPendingFiles, recoverTask, requiresDelayedCommitOutputInFileSystem, setConf, setDestFS, setOutputPath, setWorkPath, startOperation, updateCommonContext, warnOnActiveUploads

Methods inherited from class org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter

hasOutputPath

Methods inherited from class org.apache.hadoop.mapreduce.OutputCommitter

isCommitJobRepeatable, isRecoverySupported, isRecoverySupported

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

NAME

public static final String NAME

Name: "staging".

See Also:

Constant Field Values

Constructor Detail

StagingCommitter

public StagingCommitter(org.apache.hadoop.fs.Path outputPath,
                        org.apache.hadoop.mapreduce.TaskAttemptContext context)
                 throws IOException

Committer for a single task attempt.

Parameters:

outputPath - final output path

context - task context

Throws:

IOException - on a failure

Method Detail

getName

public String getName()

Description copied from class: AbstractS3ACommitter

Get the name of this committer.

Specified by:

getName in class AbstractS3ACommitter

Returns:

the committer name.

createWrappedCommitter

protected org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter createWrappedCommitter(org.apache.hadoop.mapreduce.JobContext context,
                                                                                            org.apache.hadoop.conf.Configuration conf)
                                                                                     throws IOException

Create the wrapped committer. This includes customizing its options, and setting up the destination directory.

Parameters:

context - job/task context.

conf - config

Returns:

the inner committer

Throws:

IOException - on a failure

initFileOutputCommitterOptions

protected void initFileOutputCommitterOptions(org.apache.hadoop.mapreduce.JobContext context)

Init the context config with everything needed for the file output committer. In particular, this code currently only works with commit algorithm 1.

Parameters:

context - context to configure.

toString

public String toString()

Overrides:

toString in class AbstractS3ACommitter

useUniqueFilenames

public Boolean useUniqueFilenames()

Is this committer using unique filenames?

Returns:

true if unique filenames are used.

getJobAttemptFileSystem

public org.apache.hadoop.fs.FileSystem getJobAttemptFileSystem(org.apache.hadoop.mapreduce.JobContext context)
                                                        throws IOException

Get the filesystem for the job attempt.

Parameters:

context - the context of the job. This is used to get the application attempt ID.

Returns:

the FS to store job attempt data.

Throws:

IOException - failure to create the FS.

getJobAttemptPath

public static org.apache.hadoop.fs.Path getJobAttemptPath(org.apache.hadoop.mapreduce.JobContext context,
                                                          org.apache.hadoop.fs.Path out)

Compute the path where the output of a given job attempt will be placed.

Parameters:

context - the context of the job. This is used to get the application attempt ID.

out - the output path to place these in.

Returns:

the path to store job attempt data.

getJobAttemptPath

protected org.apache.hadoop.fs.Path getJobAttemptPath(int appAttemptId)

Description copied from class: AbstractS3ACommitter

Compute the path where the output of a given job attempt will be placed.

Specified by:

getJobAttemptPath in class AbstractS3ACommitter

Parameters:

appAttemptId - the ID of the application attempt for this job.

Returns:

the path to store job attempt data.

getJobPath

protected org.apache.hadoop.fs.Path getJobPath()

Compute the path under which all job attempts will be placed.

Specified by:

getJobPath in class AbstractS3ACommitter

Returns:

the path to store job attempt data.

getTaskAttemptPath

public static org.apache.hadoop.fs.Path getTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context,
                                                           org.apache.hadoop.fs.Path out)

Compute the path where the output of a task attempt is stored until that task is committed.

Parameters:

context - the context of the task attempt.

out - The output path to put things in.

Returns:

the path where a task attempt should be stored.

getCommittedTaskPath

public org.apache.hadoop.fs.Path getCommittedTaskPath(org.apache.hadoop.mapreduce.TaskAttemptContext context)

Compute the path where the output of a committed task is stored until the entire job is committed.

Parameters:

context - the context of the task attempt

Returns:

the path where the output of a committed task is stored until the entire job is committed.

getCommittedTaskPath

protected org.apache.hadoop.fs.Path getCommittedTaskPath(int appAttemptId,
                                                         org.apache.hadoop.mapreduce.TaskAttemptContext context)

Compute the path where the output of a committed task is stored until the entire job is committed for a specific application attempt.

Parameters:

appAttemptId - the ID of the application attempt to use

context - the context of any task.

Returns:

the path where the output of a committed task is stored.

getTempTaskAttemptPath

public org.apache.hadoop.fs.Path getTempTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context)

Description copied from class: AbstractS3ACommitter

Get a temporary directory for data. When a task is aborted/cleaned up, the contents of this directory are all deleted.

Specified by:

getTempTaskAttemptPath in class AbstractS3ACommitter

Parameters:

context - task context

Returns:

a path for temporary data.

getTaskOutput

protected List<org.apache.hadoop.fs.LocatedFileStatus> getTaskOutput(org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                                              throws IOException

Lists the output of a task under the task attempt path. Subclasses can override this method to change how output files are identified.

This implementation lists the files that are direct children of the output path and filters hidden files (file names starting with '.' or '_').

The task attempt path is provided by AbstractS3ACommitter.getTaskAttemptPath(TaskAttemptContext)

Parameters:

context - this task's TaskAttemptContext

Returns:

the output files produced by this task in the task attempt path

Throws:

IOException - on a failure

getFinalKey

protected String getFinalKey(String relative,
                             org.apache.hadoop.mapreduce.JobContext context)

Returns the final S3 key for a relative path. Subclasses can override this method to upload files to a different S3 location.

This implementation concatenates the relative path with the key prefix from the output path. If CommitConstants.FS_S3A_COMMITTER_STAGING_UNIQUE_FILENAMES is set, then the task UUID is also included in the calculation

Parameters:

relative - the path of a file relative to the task attempt path

context - the JobContext or TaskAttemptContext for this job

Returns:

the S3 key where the file will be uploaded

getFinalPath

protected final org.apache.hadoop.fs.Path getFinalPath(String relative,
                                                       org.apache.hadoop.mapreduce.JobContext context)
                                                throws IOException

Returns the final S3 location for a relative path as a Hadoop Path. This is a final method that calls getFinalKey(String, JobContext) to determine the final location.

Parameters:

relative - the path of a file relative to the task attempt path

context - the JobContext or TaskAttemptContext for this job

Returns:

the S3 Path where the file will be uploaded

Throws:

IOException - IO problem

getBaseTaskAttemptPath

public org.apache.hadoop.fs.Path getBaseTaskAttemptPath(org.apache.hadoop.mapreduce.TaskAttemptContext context)

Return the local work path as the destination for writing work.

Specified by:

getBaseTaskAttemptPath in class AbstractS3ACommitter

Parameters:

context - the context of the task attempt.

Returns:

a path in the local filesystem.

getJobAttemptPath

public org.apache.hadoop.fs.Path getJobAttemptPath(org.apache.hadoop.mapreduce.JobContext context)

For a job attempt path, the staging committer returns that of the wrapped committer.

Overrides:

getJobAttemptPath in class AbstractS3ACommitter

Parameters:

context - the context of the job.

Returns:

a path in HDFS.

setupJob

public void setupJob(org.apache.hadoop.mapreduce.JobContext context)
              throws IOException

Set up the job, including calling the same method on the wrapped committer.

Overrides:

setupJob in class AbstractS3ACommitter

Parameters:

context - job context

Throws:

IOException - IO failure.

listPendingUploadsToCommit

protected AbstractS3ACommitter.ActiveCommit listPendingUploadsToCommit(CommitContext commitContext)
                                                                throws IOException

Get the list of pending uploads for this job attempt.

Specified by:

listPendingUploadsToCommit in class AbstractS3ACommitter

Parameters:

commitContext - job context

Returns:

a list of pending uploads.

Throws:

IOException - Any IO failure

listPendingUploadsToAbort

protected AbstractS3ACommitter.ActiveCommit listPendingUploadsToAbort(CommitContext commitContext)
                                                               throws IOException

Get the list of pending uploads for this job attempt, swallowing exceptions.

Parameters:

commitContext - commit context

Returns:

a list of pending uploads. If an exception was swallowed, then this may not match the actual set of pending operations

Throws:

IOException - shouldn't be raised, but retained for the compiler

listPendingUploads

protected AbstractS3ACommitter.ActiveCommit listPendingUploads(CommitContext commitContext,
                                                               boolean suppressExceptions)
                                                        throws IOException

Get the list of pending uploads for this job attempt.

Parameters:

commitContext - commit context

suppressExceptions - should exceptions be swallowed?

Returns:

a list of pending uploads. If exceptions are being swallowed, then this may not match the actual set of pending operations

Throws:

IOException - Any IO failure which wasn't swallowed.

cleanupStagingDirs

public void cleanupStagingDirs()

Description copied from class: AbstractS3ACommitter

Clean up any staging directories. IOEs must be caught and swallowed.

Specified by:

cleanupStagingDirs in class AbstractS3ACommitter

cleanup

protected void cleanup(CommitContext commitContext,
                       boolean suppressExceptions)
                throws IOException

Staging committer cleanup includes calling wrapped committer's cleanup method, and removing staging uploads path and all destination paths in the final filesystem.

Overrides:

cleanup in class AbstractS3ACommitter

Parameters:

commitContext - commit context

suppressExceptions - should exceptions be suppressed?

Throws:

IOException - IO failures if exceptions are not suppressed.

abortJobInternal

protected void abortJobInternal(CommitContext commitContext,
                                boolean suppressExceptions)
                         throws IOException

Description copied from class: AbstractS3ACommitter

The internal job abort operation; can be overridden in tests. This must clean up operations; it is called when a commit fails, as well as in an AbstractS3ACommitter.abortJob(JobContext, JobStatus.State) call. The base implementation calls AbstractS3ACommitter.cleanup(CommitContext, boolean) so cleans up the filesystems and destroys the thread pool. Subclasses must always invoke this superclass method after their own operations. Creates and closes its own commit context.

Overrides:

abortJobInternal in class AbstractS3ACommitter

Parameters:

commitContext - commit context

suppressExceptions - should exceptions be suppressed?

Throws:

IOException - any IO problem raised when suppressExceptions is false.

deleteStagingUploadsParentDirectory

protected void deleteStagingUploadsParentDirectory(org.apache.hadoop.mapreduce.JobContext context)
                                            throws IOException

Delete the multipart upload staging directory.

Parameters:

context - job context

Throws:

IOException - IO failure

deleteDestinationPaths

protected void deleteDestinationPaths(org.apache.hadoop.mapreduce.JobContext context)
                               throws IOException

Delete the working paths of a job.

1. $dest/__temporary
2. the local working directory for staged files

Does not attempt to clean up the work of the wrapped committer.

Parameters:

context - job context

Throws:

IOException - IO failure

setupTask

public void setupTask(org.apache.hadoop.mapreduce.TaskAttemptContext context)
               throws IOException

Description copied from class: AbstractS3ACommitter

Task setup. Fails if the the UUID was generated locally, and the same committer wasn't used for job setup.

Overrides:

setupTask in class AbstractS3ACommitter

Throws:

IOException

needsTaskCommit

public boolean needsTaskCommit(org.apache.hadoop.mapreduce.TaskAttemptContext context)
                        throws IOException

Specified by:

needsTaskCommit in class org.apache.hadoop.mapreduce.OutputCommitter

Throws:

IOException

commitTask

public void commitTask(org.apache.hadoop.mapreduce.TaskAttemptContext context)
                throws IOException

Specified by:

commitTask in class org.apache.hadoop.mapreduce.OutputCommitter

Throws:

IOException

commitTaskInternal

protected int commitTaskInternal(org.apache.hadoop.mapreduce.TaskAttemptContext context,
                                 List<? extends org.apache.hadoop.fs.FileStatus> taskOutput,
                                 CommitContext commitContext)
                          throws IOException

Commit the task by uploading all created files and then writing a pending entry for them.

Parameters:

context - task context

taskOutput - list of files from the output

commitContext - commit context

Returns:

number of uploads committed.

Throws:

IOException - IO Failures.

abortTask

public void abortTask(org.apache.hadoop.mapreduce.TaskAttemptContext context)
               throws IOException

Abort the task. The API specifies that the task has not yet been committed, so there are no uploads that need to be cancelled. Accordingly just delete files on the local FS, and call abortTask in the wrapped committer. Important: this may be called in the AM after a container failure. When that occurs and the failed container was on a different host in the cluster, the local files will not be deleted.

Specified by:

abortTask in class org.apache.hadoop.mapreduce.OutputCommitter

Parameters:

context - task context

Throws:

IOException - any failure

deleteTaskWorkingPathQuietly

protected void deleteTaskWorkingPathQuietly(org.apache.hadoop.mapreduce.JobContext context)

Delete the working path of a task; no-op if there is none, that is: this is a job.

Parameters:

context - job/task context

getConflictResolutionMode

public final ConflictResolution getConflictResolutionMode(org.apache.hadoop.mapreduce.JobContext context,
                                                          org.apache.hadoop.conf.Configuration fsConf)

Returns the ConflictResolution mode for this commit.

Parameters:

context - the JobContext for this commit

fsConf - filesystem config

Returns:

the ConflictResolution mode

failDestinationExists

protected org.apache.hadoop.fs.PathExistsException failDestinationExists(org.apache.hadoop.fs.Path path,
                                                                         String description)

Generate a PathExistsException because the destination exists. Lists some of the child entries first, to help diagnose the problem.

Parameters:

path - path which exists

description - description (usually task/job ID)

Returns:

an exception to throw

getConfictModeOption

public static String getConfictModeOption(org.apache.hadoop.mapreduce.JobContext context,
                                          org.apache.hadoop.conf.Configuration fsConf,
                                          String defVal)

Get the conflict mode option string.

Parameters:

context - context with the config

fsConf - filesystem config

defVal - default value.

Returns:

the trimmed configuration option, upper case.

preCommitJob

public void preCommitJob(CommitContext commitContext,
                         AbstractS3ACommitter.ActiveCommit pending)
                  throws IOException

Pre-commit actions for a job. Loads all the pending files to verify they can be loaded and parsed.

Overrides:

preCommitJob in class AbstractS3ACommitter

Parameters:

commitContext - commit context

pending - pending commits

Throws:

IOException - any failure