org.apache.hadoop.fs.s3a.commit.staging

Class PartitionedStagingCommitter

java.lang.Object

org.apache.hadoop.mapreduce.OutputCommitter

org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter

org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter

org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter

org.apache.hadoop.fs.s3a.commit.staging.PartitionedStagingCommitter

public class PartitionedStagingCommitter
extends StagingCommitter

Partitioned committer. This writes data to specific "partition" subdirectories, applying conflict resolution on a partition-by-partition basis. The existence and state of any parallel partitions for which there is no are output files are not considered in the conflict resolution. The conflict policy is

• FAIL: fail the commit if any of the partitions have data.
• APPEND: add extra data to the destination partitions.
• REPLACE: delete the destination partition in the job commit (i.e. after and only if all tasks have succeeded.

Field Summary

Fields

Modifier and Type	Field and Description
static String	NAME Name: "partitioned".

Constructor Summary

Constructors

Constructor and Description
PartitionedStagingCommitter(org.apache.hadoop.fs.Path outputPath, org.apache.hadoop.mapreduce.TaskAttemptContext context)

Method Summary

Modifier and Type	Method and Description
protected int	commitTaskInternal(org.apache.hadoop.mapreduce.TaskAttemptContext context, List<? extends org.apache.hadoop.fs.FileStatus> taskOutput) Commit the task by uploading all created files and then writing a pending entry for them.
String	getName() Get the name of this committer.
protected void	preCommitJob(org.apache.hadoop.mapreduce.JobContext context, List<SinglePendingCommit> pending) Job-side conflict resolution.
String	toString()

Methods inherited from class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter

abortJobInternal, abortPendingUploadsInCleanup, abortTask, cleanup, cleanupStagingDirs, commitTask, createWrappedCommitter, deleteDestinationPaths, deleteTaskWorkingPathQuietly, failDestinationExists, getBaseTaskAttemptPath, getCommittedTaskPath, getCommittedTaskPath, getConfictModeOption, getConflictResolutionMode, getFinalKey, getFinalPath, getJobAttemptFileSystem, getJobAttemptPath, getJobAttemptPath, getJobAttemptPath, getTaskAttemptPath, getTaskOutput, getTempTaskAttemptPath, getUploadUUID, getUploadUUID, getUUID, initFileOutputCommitterOptions, listPendingUploads, listPendingUploadsToAbort, listPendingUploadsToCommit, needsTaskCommit, setupJob, setupTask, useUniqueFilenames

Methods inherited from class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter

abortJob, abortPendingUploads, buildThreadPool, cleanupJob, commitJob, commitJobInternal, commitPendingUploads, deleteTaskAttemptPathQuietly, getCommitOperations, getConf, getDestFS, getDestinationFS, getDestS3AFS, getJobContext, getOutputPath, getRole, getTaskAttemptFilesystem, getTaskAttemptPath, getWorkPath, initOutput, jobCompleted, loadPendingsetFiles, maybeCreateSuccessMarker, maybeCreateSuccessMarkerFromCommits, maybeIgnore, maybeIgnore, recoverTask, requiresDelayedCommitOutputInFileSystem, setConf, setDestFS, setOutputPath, setWorkPath

Methods inherited from class org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter

hasOutputPath

Methods inherited from class org.apache.hadoop.mapreduce.OutputCommitter

isCommitJobRepeatable, isRecoverySupported, isRecoverySupported

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

NAME

public static final String NAME

Name: "partitioned".

See Also:

Constant Field Values

Constructor Detail

PartitionedStagingCommitter

public PartitionedStagingCommitter(org.apache.hadoop.fs.Path outputPath,
                                   org.apache.hadoop.mapreduce.TaskAttemptContext context)
                            throws IOException

Throws:

IOException

Method Detail

getName

public String getName()

Description copied from class: AbstractS3ACommitter

Get the name of this committer.

Overrides:

getName in class StagingCommitter

Returns:

the committer name.

toString

public String toString()

Overrides:

toString in class StagingCommitter

commitTaskInternal

protected int commitTaskInternal(org.apache.hadoop.mapreduce.TaskAttemptContext context,
                                 List<? extends org.apache.hadoop.fs.FileStatus> taskOutput)
                          throws IOException

Description copied from class: StagingCommitter

Commit the task by uploading all created files and then writing a pending entry for them.

Overrides:

commitTaskInternal in class StagingCommitter

Parameters:

context - task context

taskOutput - list of files from the output

Returns:

number of uploads committed.

Throws:

IOException - IO Failures.

preCommitJob

protected void preCommitJob(org.apache.hadoop.mapreduce.JobContext context,
                            List<SinglePendingCommit> pending)
                     throws IOException

Job-side conflict resolution. The partition path conflict resolution actions are:

1. FAIL: assume checking has taken place earlier; no more checks.
2. APPEND: allowed.; no need to check.
3. REPLACE deletes all existing partitions.

Overrides:

preCommitJob in class AbstractS3ACommitter

Parameters:

context - job context

pending - the pending operations

Throws:

IOException - any failure