org.apache.hadoop.fs.s3a.s3guard

Class S3Guard

java.lang.Object

org.apache.hadoop.fs.s3a.s3guard.S3Guard

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public final class S3Guard
extends Object

Logic for integrating MetadataStore with S3A.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	S3Guard.DisabledWarnLevel Warning levels to use when reporting S3Guard as disabled.
static class	S3Guard.TtlTimeProvider Runtime implementation for TTL Time Provider interface.

Field Summary

Fields

Modifier and Type	Field and Description
static boolean	DIR_MERGE_UPDATES_ALL_RECORDS_NONAUTH Hard-coded policy : false.
static String	DISABLED_LOG_MSG Format string to use when warning that S3Guard is disabled.
static String	S3GUARD_DDB_CLIENT_FACTORY_IMPL
static String	UNKNOWN_WARN_LEVEL Error string use in exception raised on an unknown log level.

Method Summary

Modifier and Type	Method and Description
static void	addAncestors(MetadataStore metadataStore, org.apache.hadoop.fs.Path qualifiedPath, ITtlTimeProvider timeProvider, BulkOperationState operationState) This adds all new ancestors of a path as directories.
static void	addMoveAncestors(MetadataStore ms, Collection<org.apache.hadoop.fs.Path> srcPaths, Collection<PathMetadata> dstMetas, org.apache.hadoop.fs.Path srcRoot, org.apache.hadoop.fs.Path srcPath, org.apache.hadoop.fs.Path dstPath, String owner) Helper method that records the move of all ancestors of a path.
static void	addMoveDir(MetadataStore ms, Collection<org.apache.hadoop.fs.Path> srcPaths, Collection<PathMetadata> dstMetas, org.apache.hadoop.fs.Path srcPath, org.apache.hadoop.fs.Path dstPath, String owner) Helper function that records the move of directory paths, adding resulting metadata to the supplied lists.
static void	addMoveFile(MetadataStore ms, Collection<org.apache.hadoop.fs.Path> srcPaths, Collection<PathMetadata> dstMetas, org.apache.hadoop.fs.Path srcPath, org.apache.hadoop.fs.Path dstPath, long size, long blockSize, String owner, String eTag, String versionId) Like addMoveDir(MetadataStore, Collection, Collection, Path, Path, String) (), but for files.
static boolean	allowAuthoritative(org.apache.hadoop.fs.Path p, S3AFileSystem fs, boolean authMetadataStore, Collection<String> authPaths) Is the path for the given FS instance authoritative?
static void	assertQualified(org.apache.hadoop.fs.Path... paths) Assert that all paths are valid.
static void	assertQualified(org.apache.hadoop.fs.Path p) Assert that the path is qualified with a host and scheme.
static org.apache.hadoop.fs.RemoteIterator<S3AFileStatus>	dirListingUnion(MetadataStore ms, org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.RemoteIterator<S3AFileStatus> backingStatuses, DirListingMetadata dirMeta, boolean isAuthoritative, ITtlTimeProvider timeProvider, Function<S3AFileStatus[],org.apache.hadoop.fs.RemoteIterator<S3AFileStatus>> toStatusItr) Given directory listing metadata from both the backing store and the MetadataStore, merge the two sources of truth to create a consistent view of the current directory contents, which can be returned to clients.
static S3AFileStatus[]	dirMetaToStatuses(DirListingMetadata dirMeta) Convert the data of a directory listing to an array of FileStatus entries.
static Collection<String>	getAuthoritativePaths(S3AFileSystem fs)
static MetadataStore	getMetadataStore(org.apache.hadoop.fs.FileSystem fs, ITtlTimeProvider ttlTimeProvider) Create a new instance of the configured MetadataStore.
static PathMetadata	getWithTtl(MetadataStore ms, org.apache.hadoop.fs.Path path, ITtlTimeProvider timeProvider, boolean needEmptyDirectoryFlag, boolean allowAuthoritative) Get a path entry provided it is not considered expired.
static BulkOperationState	initiateBulkWrite(MetadataStore metastore, BulkOperationState.OperationType operation, org.apache.hadoop.fs.Path path) Initiate a bulk write and create an operation state for it.
static boolean	isNullMetadataStore(MetadataStore ms) Although NullMetadataStore does nothing, callers may wish to avoid work (fast path) when the NullMetadataStore is in use.
static DirListingMetadata	listChildrenWithTtl(MetadataStore ms, org.apache.hadoop.fs.Path path, ITtlTimeProvider timeProvider, boolean allowAuthoritative) List children; mark the result as non-auth if the TTL has expired.
static void	logS3GuardDisabled(org.slf4j.Logger logger, String warnLevelStr, String bucket) Log that S3Guard is disabled -optionally raise an exception.
static void	makeDirsOrdered(MetadataStore ms, List<org.apache.hadoop.fs.Path> dirs, String owner, boolean authoritative, ITtlTimeProvider timeProvider) Deprecated. this is no longer called by S3AFilesystem.innerMkDirs. See: HADOOP-15079 (January 2018). It is currently retained because of its discussion in the method on atomicity and in case we need to reinstate it or adapt the current process of directory marker creation. But it is not being tested and so may age with time...consider deleting it in future if it's clear there's no need for it.
static S3AFileStatus	putAndReturn(MetadataStore ms, S3AFileStatus status, ITtlTimeProvider timeProvider) Helper function which puts a given S3AFileStatus into the MetadataStore and returns the same S3AFileStatus.
static S3AFileStatus	putAndReturn(MetadataStore ms, S3AFileStatus status, ITtlTimeProvider timeProvider, BulkOperationState operationState) Helper function which puts a given S3AFileStatus into the MetadataStore and returns the same S3AFileStatus.
static void	putAuthDirectoryMarker(MetadataStore ms, S3AFileStatus status, ITtlTimeProvider timeProvider, BulkOperationState operationState) Creates an authoritative directory marker for the store.
static void	putWithTtl(MetadataStore ms, Collection<? extends PathMetadata> fileMetas, ITtlTimeProvider timeProvider, BulkOperationState operationState) Put entries, using the time provider to set their timestamp.
static void	putWithTtl(MetadataStore ms, DirListingMetadata dirMeta, List<org.apache.hadoop.fs.Path> unchangedEntries, ITtlTimeProvider timeProvider, BulkOperationState operationState) Put a directory entry, setting the updated timestamp of the directory and its children.
static void	putWithTtl(MetadataStore ms, PathMetadata fileMeta, ITtlTimeProvider timeProvider, BulkOperationState operationState) Put an entry, using the time provider to set its timestamp.
static boolean	refreshEntry(MetadataStore metadataStore, PathMetadata pm, S3AFileStatus s3AFileStatus, ITtlTimeProvider timeProvider) We update the metastore for the specific case of S3 value == S3Guard value so as to place a more recent modtime in the store.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

S3GUARD_DDB_CLIENT_FACTORY_IMPL

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public static final String S3GUARD_DDB_CLIENT_FACTORY_IMPL

See Also:

Constant Field Values

DIR_MERGE_UPDATES_ALL_RECORDS_NONAUTH

public static final boolean DIR_MERGE_UPDATES_ALL_RECORDS_NONAUTH

Hard-coded policy : false. If true, when merging an S3 LIST with S3Guard in non-auth mode, only updated entries are added; new entries are left out. This policy choice reduces the amount of data stored in Dynamo, and hence the complexity of the merge in a non-auth listing.

See Also:

Constant Field Values

DISABLED_LOG_MSG

public static final String DISABLED_LOG_MSG

Format string to use when warning that S3Guard is disabled.

See Also:

Constant Field Values

UNKNOWN_WARN_LEVEL

public static final String UNKNOWN_WARN_LEVEL

Error string use in exception raised on an unknown log level.

See Also:

Constant Field Values

Method Detail

getMetadataStore

@Retries.OnceTranslated
public static MetadataStore getMetadataStore(org.apache.hadoop.fs.FileSystem fs,
                                                                     ITtlTimeProvider ttlTimeProvider)
                                                              throws IOException

Create a new instance of the configured MetadataStore. The returned MetadataStore will have been initialized via MetadataStore.initialize(FileSystem, ITtlTimeProvider) by this function before returning it. Callers must clean up by calling Closeable.close() when done using the MetadataStore.

Parameters:

fs - FileSystem whose Configuration specifies which implementation to use.

ttlTimeProvider -

Returns:

Reference to new MetadataStore.

Throws:

IOException - if the metadata store cannot be instantiated

refreshEntry

@Retries.RetryTranslated
public static boolean refreshEntry(MetadataStore metadataStore,
                                                            PathMetadata pm,
                                                            S3AFileStatus s3AFileStatus,
                                                            ITtlTimeProvider timeProvider)
                                                     throws IOException

We update the metastore for the specific case of S3 value == S3Guard value so as to place a more recent modtime in the store. because if not, we will continue to probe S3 whenever we look for this object, even we only do this if confident the S3 status is the same as the one in the store (i.e. it is not an older version)

Parameters:

metadataStore - MetadataStore to put() into.

pm - current data

s3AFileStatus - status to store

timeProvider - Time provider to use when writing entries

Returns:

true if the entry was updated.

Throws:

IOException - if metadata store update failed

putAndReturn

@Retries.RetryTranslated
public static S3AFileStatus putAndReturn(MetadataStore ms,
                                                                  S3AFileStatus status,
                                                                  ITtlTimeProvider timeProvider)
                                                           throws IOException

Helper function which puts a given S3AFileStatus into the MetadataStore and returns the same S3AFileStatus. Instrumentation monitors the put operation.

Parameters:

ms - MetadataStore to put() into.

status - status to store

timeProvider - Time provider to use when writing entries

Returns:

The same status as passed in

Throws:

IOException - if metadata store update failed

putAndReturn

@Retries.RetryTranslated
public static S3AFileStatus putAndReturn(MetadataStore ms,
                                                                  S3AFileStatus status,
                                                                  ITtlTimeProvider timeProvider,
                                                                  @Nullable
                                                                  BulkOperationState operationState)
                                                           throws IOException

Helper function which puts a given S3AFileStatus into the MetadataStore and returns the same S3AFileStatus. Instrumentation monitors the put operation.

Parameters:

ms - MetadataStore to put() into.

status - status to store

timeProvider - Time provider to use when writing entries

operationState - possibly-null metastore state tracker.

Returns:

The same status as passed in

Throws:

IOException - if metadata store update failed

putAuthDirectoryMarker

@Retries.RetryTranslated
public static void putAuthDirectoryMarker(MetadataStore ms,
                                                                   S3AFileStatus status,
                                                                   ITtlTimeProvider timeProvider,
                                                                   @Nullable
                                                                   BulkOperationState operationState)
                                                            throws IOException

Creates an authoritative directory marker for the store.

Parameters:

ms - MetadataStore to put() into.

status - status to store

timeProvider - Time provider to use when writing entries

operationState - possibly-null metastore state tracker.

Throws:

IOException - if metadata store update failed

initiateBulkWrite

public static BulkOperationState initiateBulkWrite(@Nullable
                                                   MetadataStore metastore,
                                                   BulkOperationState.OperationType operation,
                                                   org.apache.hadoop.fs.Path path)
                                            throws IOException

Initiate a bulk write and create an operation state for it. This may then be passed into put operations.

Parameters:

metastore - store

operation - the type of the operation.

path - path under which updates will be explicitly put.

Returns:

a store-specific state to pass into the put operations, or null

Throws:

IOException - failure

dirMetaToStatuses

public static S3AFileStatus[] dirMetaToStatuses(DirListingMetadata dirMeta)

Convert the data of a directory listing to an array of FileStatus entries. Tombstones are filtered out at this point. If the listing is null an empty array is returned.

Parameters:

dirMeta - directory listing -may be null

Returns:

a possibly-empty array of file status entries

dirListingUnion

public static org.apache.hadoop.fs.RemoteIterator<S3AFileStatus> dirListingUnion(MetadataStore ms,
                                                                                 org.apache.hadoop.fs.Path path,
                                                                                 org.apache.hadoop.fs.RemoteIterator<S3AFileStatus> backingStatuses,
                                                                                 DirListingMetadata dirMeta,
                                                                                 boolean isAuthoritative,
                                                                                 ITtlTimeProvider timeProvider,
                                                                                 Function<S3AFileStatus[],org.apache.hadoop.fs.RemoteIterator<S3AFileStatus>> toStatusItr)
                                                                          throws IOException

Given directory listing metadata from both the backing store and the MetadataStore, merge the two sources of truth to create a consistent view of the current directory contents, which can be returned to clients. Also update the MetadataStore to reflect the resulting directory listing. In not authoritative case: update file metadata if mod_time in listing of a file is greater then what is currently in the ms

Parameters:

ms - MetadataStore to use.

path - path to directory

backingStatuses - Directory listing from the backing store.

dirMeta - Directory listing from MetadataStore. May be null.

isAuthoritative - State of authoritative mode

timeProvider - Time provider to use when updating entries

toStatusItr - function to convert array of file status to RemoteIterator.

Returns:

Final result of directory listing.

Throws:

IOException - if metadata store update failed

isNullMetadataStore

public static boolean isNullMetadataStore(MetadataStore ms)

Although NullMetadataStore does nothing, callers may wish to avoid work (fast path) when the NullMetadataStore is in use.

Parameters:

ms - The MetadataStore to test

Returns:

true iff the MetadataStore is the null, or no-op, implementation.

makeDirsOrdered

@Deprecated
 @Retries.OnceExceptionsSwallowed
public static void makeDirsOrdered(MetadataStore ms,
                                                                                 List<org.apache.hadoop.fs.Path> dirs,
                                                                                 String owner,
                                                                                 boolean authoritative,
                                                                                 ITtlTimeProvider timeProvider)

Deprecated. this is no longer called by S3AFilesystem.innerMkDirs. See: HADOOP-15079 (January 2018). It is currently retained because of its discussion in the method on atomicity and in case we need to reinstate it or adapt the current process of directory marker creation. But it is not being tested and so may age with time...consider deleting it in future if it's clear there's no need for it.

Update MetadataStore to reflect creation of the given directories. If an IOException is raised while trying to update the entry, this operation catches the exception, swallows it and returns.

Parameters:

ms - MetadataStore to update.

dirs - null, or an ordered list of directories from leaf to root. E.g. if /a/ exists, and mkdirs(/a/b/c/d) is called, this list will contain [/a/b/c/d, /a/b/c, /a/b]. /a/b/c/d is an empty, dir, and the other dirs only contain their child dir.

owner - Hadoop user name.

authoritative - Whether to mark new directories as authoritative.

timeProvider - Time provider.

addMoveDir

public static void addMoveDir(MetadataStore ms,
                              Collection<org.apache.hadoop.fs.Path> srcPaths,
                              Collection<PathMetadata> dstMetas,
                              org.apache.hadoop.fs.Path srcPath,
                              org.apache.hadoop.fs.Path dstPath,
                              String owner)

Helper function that records the move of directory paths, adding resulting metadata to the supplied lists. Does not store in MetadataStore.

Parameters:

ms - MetadataStore, used to make this a no-op, when it is NullMetadataStore.

srcPaths - stores the source path here

dstMetas - stores destination metadata here

srcPath - source path to store

dstPath - destination path to store

owner - file owner to use in created records

addMoveFile

public static void addMoveFile(MetadataStore ms,
                               Collection<org.apache.hadoop.fs.Path> srcPaths,
                               Collection<PathMetadata> dstMetas,
                               org.apache.hadoop.fs.Path srcPath,
                               org.apache.hadoop.fs.Path dstPath,
                               long size,
                               long blockSize,
                               String owner,
                               String eTag,
                               String versionId)

Like addMoveDir(MetadataStore, Collection, Collection, Path, Path, String) (), but for files.

Parameters:

ms - MetadataStore, used to make this a no-op, when it is NullMetadataStore.

srcPaths - stores the source path here

dstMetas - stores destination metadata here

srcPath - source path to store

dstPath - destination path to store

size - length of file moved

blockSize - blocksize to associate with destination file

owner - file owner to use in created records

eTag - the s3 object eTag of file moved

versionId - the s3 object versionId of file moved

addMoveAncestors

public static void addMoveAncestors(MetadataStore ms,
                                    Collection<org.apache.hadoop.fs.Path> srcPaths,
                                    Collection<PathMetadata> dstMetas,
                                    org.apache.hadoop.fs.Path srcRoot,
                                    org.apache.hadoop.fs.Path srcPath,
                                    org.apache.hadoop.fs.Path dstPath,
                                    String owner)

Helper method that records the move of all ancestors of a path. In S3A, an optimization is to delete unnecessary fake directory objects if the directory is non-empty. In that case, for a nested child to move, S3A is not listing and thus moving all its ancestors (up to source root). So we take care of those inferred directories of this path explicitly. As addMoveFile(org.apache.hadoop.fs.s3a.s3guard.MetadataStore, java.util.Collection<org.apache.hadoop.fs.Path>, java.util.Collection<org.apache.hadoop.fs.s3a.s3guard.PathMetadata>, org.apache.hadoop.fs.Path, org.apache.hadoop.fs.Path, long, long, java.lang.String, java.lang.String, java.lang.String) and addMoveDir(org.apache.hadoop.fs.s3a.s3guard.MetadataStore, java.util.Collection<org.apache.hadoop.fs.Path>, java.util.Collection<org.apache.hadoop.fs.s3a.s3guard.PathMetadata>, org.apache.hadoop.fs.Path, org.apache.hadoop.fs.Path, java.lang.String), this method adds resulting metadata to the supplied lists. It does not update the MetadataStore.

Parameters:

ms - MetadataStore, no-op if it is NullMetadataStore

srcPaths - stores the source path here

dstMetas - stores destination metadata here

srcRoot - source root up to which (exclusive) should we add ancestors

srcPath - source path of the child to add ancestors

dstPath - destination path of the child to add ancestors

owner - Hadoop user name

addAncestors

@Retries.RetryTranslated
public static void addAncestors(MetadataStore metadataStore,
                                                         org.apache.hadoop.fs.Path qualifiedPath,
                                                         ITtlTimeProvider timeProvider,
                                                         @Nullable
                                                         BulkOperationState operationState)
                                                  throws IOException

This adds all new ancestors of a path as directories. This forwards to MetadataStore.addAncestors(Path, BulkOperationState).

Originally it implemented the logic to probe for an add ancestors, but with the addition of a store-specific bulk operation state it became unworkable.

Parameters:

metadataStore - store

qualifiedPath - path to update

operationState - (nullable) operational state for a bulk update

Throws:

IOException - failure

assertQualified

public static void assertQualified(org.apache.hadoop.fs.Path p)

Assert that the path is qualified with a host and scheme.

Parameters:

p - path to check

Throws:

NullPointerException - if either argument does not hold

assertQualified

public static void assertQualified(org.apache.hadoop.fs.Path... paths)

Assert that all paths are valid.

Parameters:

paths - path to check

Throws:

NullPointerException - if either argument does not hold

putWithTtl

public static void putWithTtl(MetadataStore ms,
                              DirListingMetadata dirMeta,
                              List<org.apache.hadoop.fs.Path> unchangedEntries,
                              ITtlTimeProvider timeProvider,
                              @Nullable
                              BulkOperationState operationState)
                       throws IOException

Put a directory entry, setting the updated timestamp of the directory and its children.

Parameters:

ms - metastore

dirMeta - directory

unchangedEntries - list of unchanged entries from the listing

timeProvider - nullable time provider

Throws:

IOException - failure.

putWithTtl

public static void putWithTtl(MetadataStore ms,
                              PathMetadata fileMeta,
                              @Nullable
                              ITtlTimeProvider timeProvider,
                              @Nullable
                              BulkOperationState operationState)
                       throws IOException

Put an entry, using the time provider to set its timestamp.

Parameters:

ms - metastore

fileMeta - entry to write

timeProvider - nullable time provider

operationState - nullable state for a bulk update

Throws:

IOException - failure.

putWithTtl

public static void putWithTtl(MetadataStore ms,
                              Collection<? extends PathMetadata> fileMetas,
                              @Nullable
                              ITtlTimeProvider timeProvider,
                              @Nullable
                              BulkOperationState operationState)
                       throws IOException

Put entries, using the time provider to set their timestamp.

Parameters:

ms - metastore

fileMetas - file metadata entries.

timeProvider - nullable time provider

operationState - nullable state for a bulk update

Throws:

IOException - failure.

getWithTtl

public static PathMetadata getWithTtl(MetadataStore ms,
                                      org.apache.hadoop.fs.Path path,
                                      @Nullable
                                      ITtlTimeProvider timeProvider,
                                      boolean needEmptyDirectoryFlag,
                                      boolean allowAuthoritative)
                               throws IOException

Get a path entry provided it is not considered expired. If the allowAuthoritative flag is true, return without checking for TTL expiry.

Parameters:

ms - metastore

path - path to look up.

timeProvider - nullable time provider

needEmptyDirectoryFlag - if true, implementation will return known state of directory emptiness.

allowAuthoritative - if this flag is true, the ttl won't apply to the metadata - so it will be returned regardless of it's expiry.

Returns:

the metadata or null if there as no entry.

Throws:

IOException - failure.

listChildrenWithTtl

@Retries.RetryTranslated
public static DirListingMetadata listChildrenWithTtl(MetadataStore ms,
                                                                              org.apache.hadoop.fs.Path path,
                                                                              @Nullable
                                                                              ITtlTimeProvider timeProvider,
                                                                              boolean allowAuthoritative)
                                                                       throws IOException

List children; mark the result as non-auth if the TTL has expired. If the allowAuthoritative flag is true, return without filtering or checking for TTL expiry. If false: the expiry scan takes place and the TODO: should we always purge tombstones? Even in auth?

Parameters:

ms - metastore

path - path to look up.

timeProvider - nullable time provider

allowAuthoritative - if this flag is true, the ttl won't apply to the metadata - so it will be returned regardless of it's expiry.

Returns:

the listing of entries under a path, or null if there as no entry.

Throws:

IOException - failure.

getAuthoritativePaths

public static Collection<String> getAuthoritativePaths(S3AFileSystem fs)

allowAuthoritative

public static boolean allowAuthoritative(org.apache.hadoop.fs.Path p,
                                         S3AFileSystem fs,
                                         boolean authMetadataStore,
                                         Collection<String> authPaths)

Is the path for the given FS instance authoritative?

Parameters:

p - path

fs - filesystem

authMetadataStore - is the MS authoritative.

authPaths - possibly empty list of authoritative paths

Returns:

true iff the path is authoritative

logS3GuardDisabled

public static void logS3GuardDisabled(org.slf4j.Logger logger,
                                      String warnLevelStr,
                                      String bucket)
                               throws org.apache.hadoop.util.ExitUtil.ExitException,
                                      IllegalArgumentException

Log that S3Guard is disabled -optionally raise an exception.

Parameters:

logger - Log to log to

warnLevelStr - string value of warn action.

bucket - bucket to use in log/error messages

Throws:

ExitUtil.ExitException - if s3guard was disabled and the log level is "fail"

IllegalArgumentException - unknown warning level.