org.apache.hadoop.fs.s3a.commit.files

Class PendingSet

java.lang.Object

org.apache.hadoop.fs.s3a.commit.files.PersistentCommitData<PendingSet>

org.apache.hadoop.fs.s3a.commit.files.PendingSet

All Implemented Interfaces:

Serializable, org.apache.hadoop.fs.statistics.IOStatisticsSource

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public class PendingSet
extends PersistentCommitData<PendingSet>

Persistent format for multiple pending commits. Contains 0 or more SinglePendingCommit entries; validation logic checks those values on load.

The statistics published through the IOStatisticsSource interface are the static ones marshalled with the commit data; they may be empty.

As single commits are added via add(SinglePendingCommit), any statistics from those commits are merged into the aggregate statistics, and those of the single commit cleared.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static int	VERSION Supported version value: 3.

Constructor Summary

Constructors

Constructor and Description
PendingSet()
PendingSet(int size)

Method Summary

Modifier and Type	Method and Description
void	add(SinglePendingCommit commit) Add a commit.
List<SinglePendingCommit>	getCommits()
org.apache.hadoop.fs.statistics.IOStatisticsSnapshot	getIOStatistics()
String	getJobId()
int	getVersion()
static PendingSet	load(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.FileStatus status) Load an instance from a file, then validate it.
void	putExtraData(String key, String value) Set/Update an extra data entry.
org.apache.hadoop.fs.statistics.IOStatistics	save(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.util.JsonSerialization<PendingSet> serializer) Save to a hadoop filesystem.
static org.apache.hadoop.util.JsonSerialization<PendingSet>	serializer() Get a shared JSON serializer for this class.
void	setCommits(List<SinglePendingCommit> commits)
void	setIOStatistics(org.apache.hadoop.fs.statistics.IOStatisticsSnapshot ioStatistics)
void	setJobId(String jobId)
void	setVersion(int version)
int	size() Number of commits.
byte[]	toBytes(org.apache.hadoop.util.JsonSerialization<PendingSet> serializer) Serialize to JSON and then to a byte array, after performing a preflight validation of the data to be saved.
void	validate() Validate the data: those fields which must be non empty, must be set.

Methods inherited from class org.apache.hadoop.fs.s3a.commit.files.PersistentCommitData

load, saveFile, saveToStream

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

VERSION

public static final int VERSION

Supported version value: 3. If this is changed the value of serialVersionUID will change, to avoid deserialization problems.

See Also:

Constant Field Values

Constructor Detail

PendingSet

public PendingSet()

PendingSet

public PendingSet(int size)

Method Detail

serializer

public static org.apache.hadoop.util.JsonSerialization<PendingSet> serializer()

Get a shared JSON serializer for this class.

Returns:

a serializer.

load

public static PendingSet load(org.apache.hadoop.fs.FileSystem fs,
                              org.apache.hadoop.fs.Path path,
                              @Nullable
                              org.apache.hadoop.fs.FileStatus status)
                       throws IOException

Load an instance from a file, then validate it.

Parameters:

fs - filesystem

path - path

status - status of file to load

Returns:

the loaded instance

Throws:

IOException - IO failure

ValidationFailure - if the data is invalid

add

public void add(SinglePendingCommit commit)

Add a commit.

Parameters:

commit - the single commit

validate

public void validate()
              throws ValidationFailure

Validate the data: those fields which must be non empty, must be set.

Specified by:

validate in class PersistentCommitData<PendingSet>

Throws:

ValidationFailure - if the data is invalid

toBytes

public byte[] toBytes(org.apache.hadoop.util.JsonSerialization<PendingSet> serializer)
               throws IOException

Description copied from class: PersistentCommitData

Serialize to JSON and then to a byte array, after performing a preflight validation of the data to be saved.

Specified by:

toBytes in class PersistentCommitData<PendingSet>

Parameters:

serializer - serializer to use

Returns:

the data in a persistable form.

Throws:

IOException - serialization problem or validation failure.

size

public int size()

Number of commits.

Returns:

the number of commits in this structure.

save

public org.apache.hadoop.fs.statistics.IOStatistics save(org.apache.hadoop.fs.FileSystem fs,
                                                         org.apache.hadoop.fs.Path path,
                                                         org.apache.hadoop.util.JsonSerialization<PendingSet> serializer)
                                                  throws IOException

Description copied from class: PersistentCommitData

Save to a hadoop filesystem. The destination file is overwritten, and on s3a stores the performance flag is set to turn off all existence checks and parent dir cleanup. The assumption here is: the job knows what it is doing.

Specified by:

save in class PersistentCommitData<PendingSet>

Parameters:

fs - filesystem

path - path

serializer - serializer to use

Returns:

IOStats from the output stream.

Throws:

IOException - IO exception

getVersion

public int getVersion()

Returns:

the version marker.

setVersion

public void setVersion(int version)

getCommits

public List<SinglePendingCommit> getCommits()

Returns:

commit list.

setCommits

public void setCommits(List<SinglePendingCommit> commits)

putExtraData

public void putExtraData(String key,
                         String value)

Set/Update an extra data entry.

Parameters:

key - key

value - value

getJobId

public String getJobId()

Returns:

Job ID, if known.

setJobId

public void setJobId(String jobId)

getIOStatistics

public org.apache.hadoop.fs.statistics.IOStatisticsSnapshot getIOStatistics()

setIOStatistics

public void setIOStatistics(org.apache.hadoop.fs.statistics.IOStatisticsSnapshot ioStatistics)