org.apache.hadoop.fs.s3a.s3guard

Class S3Guard

java.lang.Object

org.apache.hadoop.fs.s3a.s3guard.S3Guard

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public final class S3Guard
extends Object

Logic for integrating MetadataStore with S3A.

Field Summary

Fields

Modifier and Type	Field and Description
static String	S3GUARD_DDB_CLIENT_FACTORY_IMPL

Method Summary

Modifier and Type	Method and Description
static void	addAncestors(MetadataStore metadataStore, org.apache.hadoop.fs.Path qualifiedPath, String username)
static void	addMoveAncestors(MetadataStore ms, Collection<org.apache.hadoop.fs.Path> srcPaths, Collection<PathMetadata> dstMetas, org.apache.hadoop.fs.Path srcRoot, org.apache.hadoop.fs.Path srcPath, org.apache.hadoop.fs.Path dstPath, String owner) Helper method that records the move of all ancestors of a path.
static void	addMoveDir(MetadataStore ms, Collection<org.apache.hadoop.fs.Path> srcPaths, Collection<PathMetadata> dstMetas, org.apache.hadoop.fs.Path srcPath, org.apache.hadoop.fs.Path dstPath, String owner) Helper function that records the move of directory paths, adding resulting metadata to the supplied lists.
static void	addMoveFile(MetadataStore ms, Collection<org.apache.hadoop.fs.Path> srcPaths, Collection<PathMetadata> dstMetas, org.apache.hadoop.fs.Path srcPath, org.apache.hadoop.fs.Path dstPath, long size, long blockSize, String owner) Like addMoveDir(MetadataStore, Collection, Collection, Path, Path, String) (), but for files.
static void	assertQualified(org.apache.hadoop.fs.Path... paths) Assert that all paths are valid.
static void	assertQualified(org.apache.hadoop.fs.Path p) Assert that the path is qualified with a host and scheme.
static org.apache.hadoop.fs.FileStatus[]	dirListingUnion(MetadataStore ms, org.apache.hadoop.fs.Path path, List<org.apache.hadoop.fs.FileStatus> backingStatuses, DirListingMetadata dirMeta, boolean isAuthoritative) Given directory listing metadata from both the backing store and the MetadataStore, merge the two sources of truth to create a consistent view of the current directory contents, which can be returned to clients.
static org.apache.hadoop.fs.FileStatus[]	dirMetaToStatuses(DirListingMetadata dirMeta) Convert the data of a directory listing to an array of FileStatus entries.
static MetadataStore	getMetadataStore(org.apache.hadoop.fs.FileSystem fs) Create a new instance of the configured MetadataStore.
static boolean	isNullMetadataStore(MetadataStore ms) Although NullMetadataStore does nothing, callers may wish to avoid work (fast path) when the NullMetadataStore is in use.
static void	makeDirsOrdered(MetadataStore ms, List<org.apache.hadoop.fs.Path> dirs, String owner, boolean authoritative) Deprecated. this is no longer called by S3AFilesystem.innerMkDirs. See: HADOOP-15079 (January 2018). It is currently retained because of its discussion in the method on atomicity and in case we need to reinstate it or adapt the current process of directory marker creation. But it is not being tested and so may age with time...consider deleting it in future if it's clear there's no need for it.
static S3AFileStatus	putAndReturn(MetadataStore ms, S3AFileStatus status, S3AInstrumentation instrumentation) Helper function which puts a given S3AFileStatus into the MetadataStore and returns the same S3AFileStatus.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

S3GUARD_DDB_CLIENT_FACTORY_IMPL

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public static final String S3GUARD_DDB_CLIENT_FACTORY_IMPL

See Also:

Constant Field Values

Method Detail

getMetadataStore

@Retries.OnceTranslated
public static MetadataStore getMetadataStore(org.apache.hadoop.fs.FileSystem fs)
                                                              throws IOException

Create a new instance of the configured MetadataStore. The returned MetadataStore will have been initialized via MetadataStore.initialize(FileSystem) by this function before returning it. Callers must clean up by calling Closeable.close() when done using the MetadataStore.

Parameters:

fs - FileSystem whose Configuration specifies which implementation to use.

Returns:

Reference to new MetadataStore.

Throws:

IOException - if the metadata store cannot be instantiated

putAndReturn

public static S3AFileStatus putAndReturn(MetadataStore ms,
                                         S3AFileStatus status,
                                         S3AInstrumentation instrumentation)
                                  throws IOException

Helper function which puts a given S3AFileStatus into the MetadataStore and returns the same S3AFileStatus. Instrumentation monitors the put operation.

Parameters:

ms - MetadataStore to put() into.

status - status to store

instrumentation - instrumentation of the s3a file system

Returns:

The same status as passed in

Throws:

IOException - if metadata store update failed

dirMetaToStatuses

public static org.apache.hadoop.fs.FileStatus[] dirMetaToStatuses(DirListingMetadata dirMeta)

Convert the data of a directory listing to an array of FileStatus entries. Tombstones are filtered out at this point. If the listing is null an empty array is returned.

Parameters:

dirMeta - directory listing -may be null

Returns:

a possibly-empty array of file status entries

dirListingUnion

public static org.apache.hadoop.fs.FileStatus[] dirListingUnion(MetadataStore ms,
                                                                org.apache.hadoop.fs.Path path,
                                                                List<org.apache.hadoop.fs.FileStatus> backingStatuses,
                                                                DirListingMetadata dirMeta,
                                                                boolean isAuthoritative)
                                                         throws IOException

Given directory listing metadata from both the backing store and the MetadataStore, merge the two sources of truth to create a consistent view of the current directory contents, which can be returned to clients. Also update the MetadataStore to reflect the resulting directory listing.

Parameters:

ms - MetadataStore to use.

path - path to directory

backingStatuses - Directory listing from the backing store.

dirMeta - Directory listing from MetadataStore. May be null.

isAuthoritative - State of authoritative mode

Returns:

Final result of directory listing.

Throws:

IOException - if metadata store update failed

isNullMetadataStore

public static boolean isNullMetadataStore(MetadataStore ms)

Although NullMetadataStore does nothing, callers may wish to avoid work (fast path) when the NullMetadataStore is in use.

Parameters:

ms - The MetadataStore to test

Returns:

true iff the MetadataStore is the null, or no-op, implementation.

makeDirsOrdered

@Deprecated
 @Retries.OnceExceptionsSwallowed
public static void makeDirsOrdered(MetadataStore ms,
                                                                                 List<org.apache.hadoop.fs.Path> dirs,
                                                                                 String owner,
                                                                                 boolean authoritative)

Deprecated. this is no longer called by S3AFilesystem.innerMkDirs. See: HADOOP-15079 (January 2018). It is currently retained because of its discussion in the method on atomicity and in case we need to reinstate it or adapt the current process of directory marker creation. But it is not being tested and so may age with time...consider deleting it in future if it's clear there's no need for it.

Update MetadataStore to reflect creation of the given directories. If an IOException is raised while trying to update the entry, this operation catches the exception, swallows it and returns.

Parameters:

ms - MetadataStore to update.

dirs - null, or an ordered list of directories from leaf to root. E.g. if /a/ exists, and mkdirs(/a/b/c/d) is called, this list will contain [/a/b/c/d, /a/b/c, /a/b]. /a/b/c/d is an empty, dir, and the other dirs only contain their child dir.

owner - Hadoop user name.

authoritative - Whether to mark new directories as authoritative.

addMoveDir

public static void addMoveDir(MetadataStore ms,
                              Collection<org.apache.hadoop.fs.Path> srcPaths,
                              Collection<PathMetadata> dstMetas,
                              org.apache.hadoop.fs.Path srcPath,
                              org.apache.hadoop.fs.Path dstPath,
                              String owner)

Helper function that records the move of directory paths, adding resulting metadata to the supplied lists. Does not store in MetadataStore.

Parameters:

ms - MetadataStore, used to make this a no-op, when it is NullMetadataStore.

srcPaths - stores the source path here

dstMetas - stores destination metadata here

srcPath - source path to store

dstPath - destination path to store

owner - file owner to use in created records

addMoveFile

public static void addMoveFile(MetadataStore ms,
                               Collection<org.apache.hadoop.fs.Path> srcPaths,
                               Collection<PathMetadata> dstMetas,
                               org.apache.hadoop.fs.Path srcPath,
                               org.apache.hadoop.fs.Path dstPath,
                               long size,
                               long blockSize,
                               String owner)

Like addMoveDir(MetadataStore, Collection, Collection, Path, Path, String) (), but for files.

Parameters:

ms - MetadataStore, used to make this a no-op, when it is NullMetadataStore.

srcPaths - stores the source path here

dstMetas - stores destination metadata here

srcPath - source path to store

dstPath - destination path to store

size - length of file moved

blockSize - blocksize to associate with destination file

owner - file owner to use in created records

addMoveAncestors

public static void addMoveAncestors(MetadataStore ms,
                                    Collection<org.apache.hadoop.fs.Path> srcPaths,
                                    Collection<PathMetadata> dstMetas,
                                    org.apache.hadoop.fs.Path srcRoot,
                                    org.apache.hadoop.fs.Path srcPath,
                                    org.apache.hadoop.fs.Path dstPath,
                                    String owner)

Helper method that records the move of all ancestors of a path. In S3A, an optimization is to delete unnecessary fake directory objects if the directory is non-empty. In that case, for a nested child to move, S3A is not listing and thus moving all its ancestors (up to source root). So we take care of those inferred directories of this path explicitly. As addMoveFile(org.apache.hadoop.fs.s3a.s3guard.MetadataStore, java.util.Collection<org.apache.hadoop.fs.Path>, java.util.Collection<org.apache.hadoop.fs.s3a.s3guard.PathMetadata>, org.apache.hadoop.fs.Path, org.apache.hadoop.fs.Path, long, long, java.lang.String) and addMoveDir(org.apache.hadoop.fs.s3a.s3guard.MetadataStore, java.util.Collection<org.apache.hadoop.fs.Path>, java.util.Collection<org.apache.hadoop.fs.s3a.s3guard.PathMetadata>, org.apache.hadoop.fs.Path, org.apache.hadoop.fs.Path, java.lang.String), this method adds resulting metadata to the supplied lists. It does not store in MetadataStore.

Parameters:

ms - MetadataStore, no-op if it is NullMetadataStore

srcPaths - stores the source path here

dstMetas - stores destination metadata here

srcRoot - source root up to which (exclusive) should we add ancestors

srcPath - source path of the child to add ancestors

dstPath - destination path of the child to add ancestors

owner - Hadoop user name

addAncestors

public static void addAncestors(MetadataStore metadataStore,
                                org.apache.hadoop.fs.Path qualifiedPath,
                                String username)
                         throws IOException

Throws:

IOException

assertQualified

public static void assertQualified(org.apache.hadoop.fs.Path p)

Assert that the path is qualified with a host and scheme.

Parameters:

p - path to check

Throws:

NullPointerException - if either argument does not hold

assertQualified

public static void assertQualified(org.apache.hadoop.fs.Path... paths)

Assert that all paths are valid.

Parameters:

paths - path to check

Throws:

NullPointerException - if either argument does not hold