org.apache.hadoop.fs.s3a.commit.magic

Class InMemoryMagicCommitTracker

java.lang.Object

org.apache.hadoop.fs.s3a.commit.PutTracker

org.apache.hadoop.fs.s3a.commit.magic.MagicCommitTracker

org.apache.hadoop.fs.s3a.commit.magic.InMemoryMagicCommitTracker

public class InMemoryMagicCommitTracker
extends MagicCommitTracker

InMemoryMagicCommitTracker stores the commit data in memory. The commit data and related data stores are flushed out from the memory when the task is committed or aborted.

Field Summary

Fields inherited from class org.apache.hadoop.fs.s3a.commit.magic.MagicCommitTracker

EMPTY, LOG

Constructor Summary

Constructors

Constructor and Description
InMemoryMagicCommitTracker(org.apache.hadoop.fs.Path path, String bucket, String originalDestKey, String destKey, String pendingsetKey, WriteOperationHelper writer, PutTrackerStatistics trackerStatistics)

Method Summary

Modifier and Type	Method and Description
boolean	aboutToComplete(String uploadId, List<software.amazon.awssdk.services.s3.model.CompletedPart> parts, long bytesWritten, org.apache.hadoop.fs.statistics.IOStatistics iostatistics) Complete operation: generate the final commit data, put it.
static Map<org.apache.hadoop.fs.Path,Long>	getPathToBytesWritten()
static Map<String,List<SinglePendingCommit>>	getTaskAttemptIdToMpuMetadata()
static Map<String,List<org.apache.hadoop.fs.Path>>	getTaskAttemptIdToPath()
String	toString()

Methods inherited from class org.apache.hadoop.fs.s3a.commit.magic.MagicCommitTracker

getBucket, getOriginalDestKey, getPath, getPendingPartKey, getTrackerStatistics, getWriter, initialize, outputImmediatelyVisible

Methods inherited from class org.apache.hadoop.fs.s3a.commit.PutTracker

getDestKey

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

InMemoryMagicCommitTracker

public InMemoryMagicCommitTracker(org.apache.hadoop.fs.Path path,
                                  String bucket,
                                  String originalDestKey,
                                  String destKey,
                                  String pendingsetKey,
                                  WriteOperationHelper writer,
                                  PutTrackerStatistics trackerStatistics)

Method Detail

aboutToComplete

public boolean aboutToComplete(String uploadId,
                               List<software.amazon.awssdk.services.s3.model.CompletedPart> parts,
                               long bytesWritten,
                               org.apache.hadoop.fs.statistics.IOStatistics iostatistics)
                        throws IOException

Description copied from class: MagicCommitTracker

Complete operation: generate the final commit data, put it.

Specified by:

aboutToComplete in class MagicCommitTracker

Parameters:

uploadId - Upload ID

parts - list of parts

bytesWritten - bytes written

iostatistics - nullable IO statistics

Returns:

false, indicating that the commit must fail.

Throws:

IOException - any IO problem.

toString

public String toString()

Overrides:

toString in class MagicCommitTracker

getTaskAttemptIdToMpuMetadata

public static Map<String,List<SinglePendingCommit>> getTaskAttemptIdToMpuMetadata()

getPathToBytesWritten

public static Map<org.apache.hadoop.fs.Path,Long> getPathToBytesWritten()

getTaskAttemptIdToPath

public static Map<String,List<org.apache.hadoop.fs.Path>> getTaskAttemptIdToPath()