org.apache.hadoop.fs.azurebfs.commit

Class AbfsManifestStoreOperations

java.lang.Object

org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.ManifestStoreOperations

org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.ManifestStoreOperationsThroughFileSystem

org.apache.hadoop.fs.azurebfs.commit.AbfsManifestStoreOperations

All Implemented Interfaces:

Closeable, AutoCloseable

@InterfaceAudience.LimitedPrivate(value="mapreduce")
 @InterfaceStability.Unstable
public class AbfsManifestStoreOperations
extends org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.ManifestStoreOperationsThroughFileSystem

Extension of StoreOperationsThroughFileSystem with ABFS awareness. Purely for use by jobs committing work through the manifest committer. The AzureManifestCommitterFactory will configure this as the binding to the FS. ADLS Gen2 stores support etag-recovery on renames, but not WASB stores.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.ManifestStoreOperations

org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.ManifestStoreOperations.CommitFileResult

Field Summary

Fields

Modifier and Type	Field and Description
static String	NAME Classname, which can be declared in jpb configurations.

Constructor Summary

Constructors

Constructor and Description
AbfsManifestStoreOperations()

Method Summary

Modifier and Type	Method and Description
void	bindToFileSystem(org.apache.hadoop.fs.FileSystem filesystem, org.apache.hadoop.fs.Path path) Bind to the store.
org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.ManifestStoreOperations.CommitFileResult	commitFile(org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.FileEntry entry) Commit a file through an internal ABFS operation.
AzureBlobFileSystem	getFileSystem()
boolean	storePreservesEtagsThroughRenames(org.apache.hadoop.fs.Path path) Etags are preserved through Gen2 stores, but not wasb stores.
boolean	storeSupportsResilientCommit() Resilient commits available on hierarchical stores.

Methods inherited from class org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.ManifestStoreOperationsThroughFileSystem

close, delete, deleteRecursive, getFileStatus, isFile, listStatusIterator, loadTaskManifest, mkdirs, msync, renameFile, save

Methods inherited from class org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.ManifestStoreOperations

deleteFile, getEtag, renameDir

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

NAME

public static final String NAME

Classname, which can be declared in jpb configurations.

Constructor Detail

AbfsManifestStoreOperations

public AbfsManifestStoreOperations()

Method Detail

getFileSystem

public AzureBlobFileSystem getFileSystem()

Overrides:

getFileSystem in class org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.ManifestStoreOperationsThroughFileSystem

bindToFileSystem

public void bindToFileSystem(org.apache.hadoop.fs.FileSystem filesystem,
                             org.apache.hadoop.fs.Path path)
                      throws IOException

Bind to the store.

Overrides:

bindToFileSystem in class org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.ManifestStoreOperationsThroughFileSystem

Parameters:

filesystem - FS.

path - path to work under

Throws:

IOException - binding problems.

storePreservesEtagsThroughRenames

public boolean storePreservesEtagsThroughRenames(org.apache.hadoop.fs.Path path)

Etags are preserved through Gen2 stores, but not wasb stores.

Overrides:

storePreservesEtagsThroughRenames in class org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.ManifestStoreOperationsThroughFileSystem

Parameters:

path - path to probe.

Returns:

true if this store preserves etags.

storeSupportsResilientCommit

public boolean storeSupportsResilientCommit()

Resilient commits available on hierarchical stores.

Overrides:

storeSupportsResilientCommit in class org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.ManifestStoreOperations

Returns:

true if the FS can use etags on renames.

commitFile

public org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.ManifestStoreOperations.CommitFileResult commitFile(org.apache.hadoop.mapreduce.lib.output.committer.manifest.files.FileEntry entry)
                                                                                                                   throws IOException

Commit a file through an internal ABFS operation. If resilient commit is unavailable, invokes the superclass, which will raise an UnsupportedOperationException

Overrides:

commitFile in class org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.ManifestStoreOperations

Parameters:

entry - entry to commit

Returns:

the outcome

Throws:

IOException - any failure in resilient commit.

UnsupportedOperationException - if not available.