org.apache.hadoop.fs.s3a.commit.files

Class PersistentCommitData<T extends PersistentCommitData>

java.lang.Object

org.apache.hadoop.fs.s3a.commit.files.PersistentCommitData<T>

All Implemented Interfaces:

Serializable, org.apache.hadoop.fs.statistics.IOStatisticsSource

Direct Known Subclasses:

PendingSet, SinglePendingCommit, SuccessData

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public abstract class PersistentCommitData<T extends PersistentCommitData>
extends Object
implements Serializable, org.apache.hadoop.fs.statistics.IOStatisticsSource

Class for single/multiple commit data structures. The mapreduce hierarchy AbstractManifestData is a fork of this; the Success data JSON format must stay compatible

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static int	VERSION Supported version value: 2.

Constructor Summary

Constructors

Constructor and Description
PersistentCommitData()

Method Summary

Modifier and Type	Method and Description
static <T extends PersistentCommitData> T	load(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.FileStatus status, org.apache.hadoop.util.JsonSerialization<T> serializer) Load an instance from a status, then validate it.
abstract org.apache.hadoop.fs.statistics.IOStatistics	save(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.util.JsonSerialization<T> serializer) Save to a hadoop filesystem.
static <T extends PersistentCommitData> org.apache.hadoop.fs.statistics.IOStatistics	saveFile(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, T instance, org.apache.hadoop.util.JsonSerialization<T> serializer, boolean performance) Save to a file.
static <T extends PersistentCommitData> org.apache.hadoop.fs.statistics.IOStatistics	saveToStream(org.apache.hadoop.fs.Path path, T instance, org.apache.hadoop.fs.FSDataOutputStreamBuilder builder, org.apache.hadoop.util.JsonSerialization<T> serializer) Save to a file.
abstract byte[]	toBytes(org.apache.hadoop.util.JsonSerialization<T> serializer) Serialize to JSON and then to a byte array, after performing a preflight validation of the data to be saved.
abstract void	validate() Validate the data: those fields which must be non empty, must be set.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.fs.statistics.IOStatisticsSource

getIOStatistics

Field Detail

VERSION

public static final int VERSION

Supported version value: 2. If this is changed the value of serialVersionUID will change, to avoid deserialization problems.

See Also:

Constant Field Values

Constructor Detail

PersistentCommitData

public PersistentCommitData()

Method Detail

validate

public abstract void validate()
                       throws ValidationFailure

Validate the data: those fields which must be non empty, must be set.

Throws:

ValidationFailure - if the data is invalid

toBytes

public abstract byte[] toBytes(org.apache.hadoop.util.JsonSerialization<T> serializer)
                        throws IOException

Serialize to JSON and then to a byte array, after performing a preflight validation of the data to be saved.

Parameters:

serializer - serializer to use

Returns:

the data in a persistable form.

Throws:

IOException - serialization problem or validation failure.

save

public abstract org.apache.hadoop.fs.statistics.IOStatistics save(org.apache.hadoop.fs.FileSystem fs,
                                                                  org.apache.hadoop.fs.Path path,
                                                                  org.apache.hadoop.util.JsonSerialization<T> serializer)
                                                           throws IOException

Save to a hadoop filesystem. The destination file is overwritten, and on s3a stores the performance flag is set to turn off all existence checks and parent dir cleanup. The assumption here is: the job knows what it is doing.

Parameters:

fs - filesystem

path - path

serializer - serializer to use

Returns:

IOStats from the output stream.

Throws:

IOException - IO exception

load

public static <T extends PersistentCommitData> T load(org.apache.hadoop.fs.FileSystem fs,
                                                      org.apache.hadoop.fs.FileStatus status,
                                                      org.apache.hadoop.util.JsonSerialization<T> serializer)
                                               throws IOException

Load an instance from a status, then validate it. This uses the openFile() API, which S3A supports for faster load and declaring sequential access, always

Type Parameters:

T - type of persistent format

Parameters:

fs - filesystem

status - status of file to load

serializer - serializer to use

Returns:

the loaded instance

Throws:

IOException - IO failure

ValidationFailure - if the data is invalid

saveFile

public static <T extends PersistentCommitData> org.apache.hadoop.fs.statistics.IOStatistics saveFile(org.apache.hadoop.fs.FileSystem fs,
                                                                                                     org.apache.hadoop.fs.Path path,
                                                                                                     T instance,
                                                                                                     org.apache.hadoop.util.JsonSerialization<T> serializer,
                                                                                                     boolean performance)
                                                                                              throws IOException

Save to a file. This uses the createFile() API, which S3A supports for faster load and declaring sequential access, always

Type Parameters:

T - type of persistent format

Parameters:

fs - filesystem

path - path to save to

instance - data to save

serializer - serializer to use

performance - skip all safety check on the write

Returns:

any IOStatistics from the output stream, or null

Throws:

IOException - IO failure

saveToStream

public static <T extends PersistentCommitData> org.apache.hadoop.fs.statistics.IOStatistics saveToStream(org.apache.hadoop.fs.Path path,
                                                                                                         T instance,
                                                                                                         org.apache.hadoop.fs.FSDataOutputStreamBuilder builder,
                                                                                                         org.apache.hadoop.util.JsonSerialization<T> serializer)
                                                                                                  throws IOException

Save to a file. This uses the createFile() API, which S3A supports for faster load and declaring sequential access, always

Type Parameters:

T - type of persistent format

Parameters:

path - path to save to (used for logging)

instance - data to save

builder - builder already prepared for the write

serializer - serializer to use

Returns:

any IOStatistics from the output stream, or null

Throws:

IOException - IO failure