org.apache.hadoop.fs.s3a.commit.staging

Class PartitionedStagingCommitter

java.lang.Object

org.apache.hadoop.mapreduce.OutputCommitter

org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter

org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter

org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter

org.apache.hadoop.fs.s3a.commit.staging.PartitionedStagingCommitter

All Implemented Interfaces:

org.apache.hadoop.fs.statistics.IOStatisticsSource

public class PartitionedStagingCommitter
extends StagingCommitter

Partitioned committer. This writes data to specific "partition" subdirectories, applying conflict resolution on a partition-by-partition basis. The existence and state of any parallel partitions for which there is no are output files are not considered in the conflict resolution. The conflict policy is

• FAIL: fail the commit if any of the partitions have data.
• APPEND: add extra data to the destination partitions.
• REPLACE: delete the destination partition in the job commit (i.e. after and only if all tasks have succeeded.

To determine the paths, the precommit process actually has to read in all source files, independently of the final commit phase. This is inefficient, though some parallelization here helps.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter

AbstractS3ACommitter.ActiveCommit, AbstractS3ACommitter.JobUUIDSource

Field Summary

Fields

Modifier and Type	Field and Description
static String	NAME Name: "partitioned".

Fields inherited from class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter

E_SELF_GENERATED_JOB_UUID, THREAD_PREFIX

Constructor Summary

Constructors

Constructor and Description
PartitionedStagingCommitter(org.apache.hadoop.fs.Path outputPath, org.apache.hadoop.mapreduce.TaskAttemptContext context)

Method Summary

Modifier and Type	Method and Description
protected int	commitTaskInternal(org.apache.hadoop.mapreduce.TaskAttemptContext context, List<? extends org.apache.hadoop.fs.FileStatus> taskOutput, CommitContext commitContext) Commit the task by uploading all created files and then writing a pending entry for them.
String	getName() Get the name of this committer.
void	preCommitJob(CommitContext commitContext, AbstractS3ACommitter.ActiveCommit pending) All Job-side conflict resolution.
String	toString()

Methods inherited from class org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter

abortJobInternal, abortTask, cleanup, cleanupStagingDirs, commitTask, createWrappedCommitter, deleteDestinationPaths, deleteStagingUploadsParentDirectory, deleteTaskWorkingPathQuietly, failDestinationExists, getBaseTaskAttemptPath, getCommittedTaskPath, getCommittedTaskPath, getConfictModeOption, getConflictResolutionMode, getFinalKey, getFinalPath, getJobAttemptFileSystem, getJobAttemptPath, getJobAttemptPath, getJobAttemptPath, getJobPath, getTaskAttemptPath, getTaskOutput, getTempTaskAttemptPath, initFileOutputCommitterOptions, listPendingUploads, listPendingUploadsToAbort, listPendingUploadsToCommit, needsTaskCommit, setupJob, setupTask, useUniqueFilenames

Methods inherited from class org.apache.hadoop.fs.s3a.commit.AbstractS3ACommitter

abortJob, abortPendingUploads, abortPendingUploads, abortPendingUploadsInCleanup, buildJobUUID, cleanupJob, commitJob, commitJobInternal, commitPendingUploads, deleteTaskAttemptPathQuietly, getAuditSpanSource, getCommitOperations, getConf, getDestFS, getDestinationFS, getDestS3AFS, getIOStatistics, getJobContext, getOutputPath, getRole, getTaskAttemptFilesystem, getTaskAttemptPath, getUUID, getUUIDSource, getWorkPath, initiateJobOperation, initiateTaskOperation, initOutput, jobCompleted, maybeCreateSuccessMarker, maybeCreateSuccessMarkerFromCommits, maybeIgnore, maybeIgnore, precommitCheckPendingFiles, recoverTask, requiresDelayedCommitOutputInFileSystem, setConf, setDestFS, setOutputPath, setWorkPath, startOperation, updateCommonContext, warnOnActiveUploads

Methods inherited from class org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter

hasOutputPath

Methods inherited from class org.apache.hadoop.mapreduce.OutputCommitter

isCommitJobRepeatable, isRecoverySupported, isRecoverySupported

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

NAME

public static final String NAME

Name: "partitioned".

See Also:

Constant Field Values

Constructor Detail

PartitionedStagingCommitter

public PartitionedStagingCommitter(org.apache.hadoop.fs.Path outputPath,
                                   org.apache.hadoop.mapreduce.TaskAttemptContext context)
                            throws IOException

Throws:

IOException

Method Detail

getName

public String getName()

Description copied from class: AbstractS3ACommitter

Get the name of this committer.

Overrides:

getName in class StagingCommitter

Returns:

the committer name.

toString

public String toString()

Overrides:

toString in class StagingCommitter

commitTaskInternal

protected int commitTaskInternal(org.apache.hadoop.mapreduce.TaskAttemptContext context,
                                 List<? extends org.apache.hadoop.fs.FileStatus> taskOutput,
                                 CommitContext commitContext)
                          throws IOException

Description copied from class: StagingCommitter

Commit the task by uploading all created files and then writing a pending entry for them.

Overrides:

commitTaskInternal in class StagingCommitter

Parameters:

context - task context

taskOutput - list of files from the output

commitContext - commit context

Returns:

number of uploads committed.

Throws:

IOException - IO Failures.

preCommitJob

public void preCommitJob(CommitContext commitContext,
                         AbstractS3ACommitter.ActiveCommit pending)
                  throws IOException

All Job-side conflict resolution. The partition path conflict resolution actions are:

1. FAIL: assume checking has taken place earlier; no more checks.
2. APPEND: allowed.; no need to check.
3. REPLACE deletes all existing partitions.

Overrides:

preCommitJob in class StagingCommitter

Parameters:

commitContext - commit context

pending - the pending operations

Throws:

IOException - any failure