org.apache.hadoop.fs.s3a.s3guard

Class DynamoDBMetadataStore

java.lang.Object

org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore

All Implemented Interfaces:

Closeable, AutoCloseable, AWSPolicyProvider, MetadataStore

@InterfaceAudience.Private
 @InterfaceStability.Evolving
public class DynamoDBMetadataStore
extends Object
implements MetadataStore, AWSPolicyProvider

DynamoDBMetadataStore is a MetadataStore that persists file system metadata to DynamoDB. The current implementation uses a schema consisting of a single table. The name of the table can be configured by config key Constants.S3GUARD_DDB_TABLE_NAME_KEY. By default, it matches the name of the S3 bucket. Each item in the table represents a single directory or file. Its path is split into separate table attributes:

• parent (absolute path of the parent, with bucket name inserted as first path component).
• child (path of that specific child, relative to parent).
• optional boolean attribute tracking whether the path is a directory. Absence or a false value indicates the path is a file.
• optional long attribute revealing modification time of file. This attribute is meaningful only to file items.
• optional long attribute revealing file length. This attribute is meaningful only to file items.
• optional long attribute revealing block size of the file. This attribute is meaningful only to file items.
• optional string attribute tracking the s3 eTag of the file. May be absent if the metadata was entered with a version of S3Guard before this was tracked. This attribute is meaningful only to file items.
• optional string attribute tracking the s3 versionId of the file. May be absent if the metadata was entered with a version of S3Guard before this was tracked. This attribute is meaningful only to file items.

The DynamoDB partition key is the parent, and the range key is the child. To allow multiple buckets to share the same DynamoDB table, the bucket name is treated as the root directory. For example, assume the consistent store contains metadata representing this file system structure:

 s3a://bucket/dir1
 |-- dir2
 |   |-- file1
 |   `-- file2
 `-- dir3
     |-- dir4
     |   `-- file3
     |-- dir5
     |   `-- file4
     `-- dir6
 

This is persisted to a single DynamoDB table as:

 ====================================================================================
 | parent                 | child | is_dir | mod_time | len | etag | ver_id |  ...  |
 ====================================================================================
 | /bucket                | dir1  | true   |          |     |      |        |       |
 | /bucket/dir1           | dir2  | true   |          |     |      |        |       |
 | /bucket/dir1           | dir3  | true   |          |     |      |        |       |
 | /bucket/dir1/dir2      | file1 |        |   100    | 111 | abc  |  mno   |       |
 | /bucket/dir1/dir2      | file2 |        |   200    | 222 | def  |  pqr   |       |
 | /bucket/dir1/dir3      | dir4  | true   |          |     |      |        |       |
 | /bucket/dir1/dir3      | dir5  | true   |          |     |      |        |       |
 | /bucket/dir1/dir3/dir4 | file3 |        |   300    | 333 | ghi  |  stu   |       |
 | /bucket/dir1/dir3/dir5 | file4 |        |   400    | 444 | jkl  |  vwx   |       |
 | /bucket/dir1/dir3      | dir6  | true   |          |     |      |        |       |
 ====================================================================================
 

This choice of schema is efficient for read access patterns. get(Path) can be served from a single item lookup. listChildren(Path) can be served from a query against all rows matching the parent (the partition key) and the returned list is guaranteed to be sorted by child (the range key). Tracking whether or not a path is a directory helps prevent unnecessary queries during traversal of an entire sub-tree. Some mutating operations, notably MetadataStore.deleteSubtree(Path, BulkOperationState) and MetadataStore.move(Collection, Collection, BulkOperationState) are less efficient with this schema. They require mutating multiple items in the DynamoDB table. By default, DynamoDB access is performed within the same AWS region as the S3 bucket that hosts the S3A instance. During initialization, it checks the location of the S3 bucket and creates a DynamoDB client connected to the same region. The region may also be set explicitly by setting the config parameter fs.s3a.s3guard.ddb.region to the corresponding region.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.fs.s3a.s3guard.MetadataStore

MetadataStore.PruneMode

Nested classes/interfaces inherited from interface org.apache.hadoop.fs.s3a.auth.delegation.AWSPolicyProvider

AWSPolicyProvider.AccessLevel

Field Summary

Fields

Modifier and Type	Field and Description
static String	E_ON_DEMAND_NO_SET_CAPACITY
static org.slf4j.Logger	LOG
static org.slf4j.Logger	OPERATIONS_LOG A log of all state changing operations to the store; only updated at debug level.
static String	OPERATIONS_LOG_NAME Name of the operations log.
static int	VERSION Current version number.
static String	VERSION_MARKER_ITEM_NAME parent/child name to use in the version marker.
static String	VERSION_MARKER_TAG_NAME parent/child name to use in the version marker.

Constructor Summary

Constructors

Constructor and Description
DynamoDBMetadataStore()

Method Summary

Modifier and Type	Method and Description
void	addAncestors(org.apache.hadoop.fs.Path qualifiedPath, BulkOperationState operationState) This adds all new ancestors of a path as directories.
void	close()
void	delete(org.apache.hadoop.fs.Path path, BulkOperationState operationState) Deletes exactly one path, leaving a tombstone to prevent lingering, inconsistent copies of it from being listed.
void	deletePaths(Collection<org.apache.hadoop.fs.Path> paths, BulkOperationState operationState) Delete the paths.
void	deleteSubtree(org.apache.hadoop.fs.Path path, BulkOperationState operationState) Deletes the entire sub-tree rooted at the given path, leaving tombstones to prevent lingering, inconsistent copies of it from being listed.
void	destroy() Destroy all resources associated with the metadata store.
void	forgetMetadata(org.apache.hadoop.fs.Path path) Removes the record of exactly one path.
DDBPathMetadata	get(org.apache.hadoop.fs.Path path) Gets metadata for a path.
DDBPathMetadata	get(org.apache.hadoop.fs.Path path, boolean wantEmptyDirectoryFlag) Gets metadata for a path.
com.amazonaws.services.dynamodbv2.AmazonDynamoDB	getAmazonDynamoDB()
long	getBatchWriteCapacityExceededCount()
Map<String,String>	getDiagnostics() Get any diagnostics information from a store, as a list of (key, value) tuples for display.
MetastoreInstrumentation	getInstrumentation() Get any S3GuardInstrumentation for this store...must not be null.
Invoker	getInvoker() Get the operation invoker for write operations.
long	getReadThrottleEventCount() Get the count of read throttle events.
long	getScanThrottleEventCount() Get the count of scan throttle events.
protected DynamoDBMetadataStoreTableManager	getTableHandler()
String	getTableName()
long	getWriteThrottleEventCount() Get the count of write throttle events.
void	initialize(org.apache.hadoop.conf.Configuration config, ITtlTimeProvider ttlTp) Performs one-time initialization of the metadata store via configuration.
void	initialize(org.apache.hadoop.fs.FileSystem fs, ITtlTimeProvider ttlTp) Performs one-time initialization of the metadata store.
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.AncestorState	initiateBulkWrite(BulkOperationState.OperationType operation, org.apache.hadoop.fs.Path dest) Initiate a bulk update and create an operation state for it.
RenameTracker	initiateRenameOperation(StoreContext storeContext, org.apache.hadoop.fs.Path source, S3AFileStatus sourceStatus, org.apache.hadoop.fs.Path dest) Initiate the rename operation by creating the tracker for the filesystem to keep up to date with state changes in the S3A bucket.
List<RoleModel.Statement>	listAWSPolicyRules(Set<AWSPolicyProvider.AccessLevel> access) The administrative policy includes all DDB table operations; application access is restricted to those operations S3Guard operations require when working with data in a guarded bucket.
DirListingMetadata	listChildren(org.apache.hadoop.fs.Path path) Lists metadata for all direct children of a path.
int	markAsAuthoritative(org.apache.hadoop.fs.Path dest, BulkOperationState operationState) Mark the directories instantiated under the destination path as authoritative.
void	move(Collection<org.apache.hadoop.fs.Path> pathsToDelete, Collection<PathMetadata> pathsToCreate, BulkOperationState operationState) Record the effects of a FileSystem.rename(Path, Path) in the MetadataStore.
void	prune(MetadataStore.PruneMode pruneMode, long cutoff) Prune method with two modes of operation: MetadataStore.PruneMode.ALL_BY_MODTIME Clear any metadata older than a specified mod_time from the store.
long	prune(MetadataStore.PruneMode pruneMode, long cutoff, String keyPrefix) Prune files, in batches.
void	put(Collection<? extends PathMetadata> metas, BulkOperationState operationState) Saves metadata for any number of paths.
void	put(DirListingMetadata meta, List<org.apache.hadoop.fs.Path> unchangedEntries, BulkOperationState operationState) Save directory listing metadata.
void	put(PathMetadata meta) Saves metadata for exactly one path.
void	put(PathMetadata meta, BulkOperationState operationState) Saves metadata for exactly one path, potentially using any bulk operation state to eliminate duplicate work.
void	setTtlTimeProvider(ITtlTimeProvider ttlTimeProvider) The TtlTimeProvider has to be set during the initialization for the metadatastore, but this method can be used for testing, and change the instance during runtime.
String	toString()
void	updateParameters(Map<String,String> parameters) Tune/update parameters for an existing table.
<T> Iterable<T>	wrapWithRetries(Iterable<T> source) Wrap an iterator returned from any scan with a retrying one.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

OPERATIONS_LOG_NAME

public static final String OPERATIONS_LOG_NAME

Name of the operations log.

See Also:

Constant Field Values

OPERATIONS_LOG

public static final org.slf4j.Logger OPERATIONS_LOG

A log of all state changing operations to the store; only updated at debug level.

VERSION_MARKER_ITEM_NAME

public static final String VERSION_MARKER_ITEM_NAME

parent/child name to use in the version marker.

See Also:

Constant Field Values

VERSION_MARKER_TAG_NAME

public static final String VERSION_MARKER_TAG_NAME

parent/child name to use in the version marker.

See Also:

Constant Field Values

VERSION

public static final int VERSION

Current version number.

See Also:

Constant Field Values

E_ON_DEMAND_NO_SET_CAPACITY

public static final String E_ON_DEMAND_NO_SET_CAPACITY

See Also:

Constant Field Values

Constructor Detail

DynamoDBMetadataStore

public DynamoDBMetadataStore()

Method Detail

initialize

@Retries.OnceRaw
public void initialize(org.apache.hadoop.fs.FileSystem fs,
                                        ITtlTimeProvider ttlTp)
                                 throws IOException

Performs one-time initialization of the metadata store.. The credentials for authenticating with S3 are requested from the FS via S3AFileSystem.shareCredentials(String); this will increment the reference counter of these credentials.

Specified by:

initialize in interface MetadataStore

Parameters:

fs - S3AFileSystem associated with the MetadataStore

ttlTp - the time provider to use for metadata expiry

Throws:

IOException - on a failure

initialize

@Retries.OnceRaw
public void initialize(org.apache.hadoop.conf.Configuration config,
                                        ITtlTimeProvider ttlTp)
                                 throws IOException

Performs one-time initialization of the metadata store via configuration. This initialization depends on the configuration object to get AWS credentials, DynamoDBFactory implementation class, DynamoDB endpoints, DynamoDB table names etc. After initialization, this metadata store does not explicitly relate to any S3 bucket, which be nonexistent. This is used to operate the metadata store directly beyond the scope of the S3AFileSystem integration, e.g. command line tools. Generally, callers should use MetadataStore.initialize(FileSystem, ITtlTimeProvider) with an initialized S3AFileSystem instance. Without a filesystem to act as a reference point, the configuration itself must declare the table name and region in the Constants.S3GUARD_DDB_TABLE_NAME_KEY and Constants.S3GUARD_DDB_REGION_KEY respectively. It also creates a new credential provider list from the configuration, using the base fs.s3a.* options, as there is no bucket to infer per-bucket settings from.

Specified by:

initialize in interface MetadataStore

Parameters:

config - Configuration.

ttlTp - the time provider to use for metadata expiry

Throws:

IOException - if there is an error

IllegalArgumentException - if the configuration is incomplete

See Also:

MetadataStore.initialize(FileSystem, ITtlTimeProvider)

delete

@Retries.RetryTranslated
public void delete(org.apache.hadoop.fs.Path path,
                                            BulkOperationState operationState)
                                     throws IOException

Description copied from interface: MetadataStore

Deletes exactly one path, leaving a tombstone to prevent lingering, inconsistent copies of it from being listed. Deleting an entry with a tombstone needs a S3Guard.TtlTimeProvider because the lastUpdated field of the record has to be updated to

now

.

Specified by:

delete in interface MetadataStore

Parameters:

path - the path to delete

operationState - (nullable) operational state for a bulk update

Throws:

IOException - if there is an error

forgetMetadata

@Retries.RetryTranslated
public void forgetMetadata(org.apache.hadoop.fs.Path path)
                                             throws IOException

Description copied from interface: MetadataStore

Removes the record of exactly one path. Does not leave a tombstone (see MetadataStore.delete(Path, BulkOperationState). It is currently intended for testing only, and a need to use it as part of normal FileSystem usage is not anticipated.

Specified by:

forgetMetadata in interface MetadataStore

Parameters:

path - the path to delete

Throws:

IOException - if there is an error

deleteSubtree

@Retries.RetryTranslated
public void deleteSubtree(org.apache.hadoop.fs.Path path,
                                                   BulkOperationState operationState)
                                            throws IOException

Description copied from interface: MetadataStore

Deletes the entire sub-tree rooted at the given path, leaving tombstones to prevent lingering, inconsistent copies of it from being listed. In addition to affecting future calls to MetadataStore.get(Path), implementations must also update any stored DirListingMetadata objects which track the parent of this file. Deleting a subtree with a tombstone needs a S3Guard.TtlTimeProvider because the lastUpdated field of all records have to be updated to

now

.

Specified by:

deleteSubtree in interface MetadataStore

Parameters:

path - the root of the sub-tree to delete

operationState - (nullable) operational state for a bulk update

Throws:

IOException - if there is an error

deletePaths

@Retries.RetryTranslated
public void deletePaths(Collection<org.apache.hadoop.fs.Path> paths,
                                                 BulkOperationState operationState)
                                          throws IOException

Description copied from interface: MetadataStore

Delete the paths. There's no attempt to order the paths: they are deleted in the order passed in.

Specified by:

deletePaths in interface MetadataStore

Parameters:

paths - paths to delete.

operationState - Nullable operation state

Throws:

IOException - failure

get

@Retries.RetryTranslated
public DDBPathMetadata get(org.apache.hadoop.fs.Path path)
                                             throws IOException

Description copied from interface: MetadataStore

Gets metadata for a path.

Specified by:

get in interface MetadataStore

Parameters:

path - the path to get

Returns:

metadata for path, null if not found

Throws:

IOException - if there is an error

get

@Retries.RetryTranslated
public DDBPathMetadata get(org.apache.hadoop.fs.Path path,
                                                    boolean wantEmptyDirectoryFlag)
                                             throws IOException

Description copied from interface: MetadataStore

Gets metadata for a path. Alternate method that includes a hint whether or not the MetadataStore should do work to compute the value for PathMetadata.isEmptyDirectory(). Since determining emptiness may be an expensive operation, this can save wasted work.

Specified by:

get in interface MetadataStore

Parameters:

path - the path to get

wantEmptyDirectoryFlag - Set to true to give a hint to the MetadataStore that it should try to compute the empty directory flag.

Returns:

metadata for path, null if not found

Throws:

IOException - if there is an error

listChildren

@Retries.RetryTranslated
public DirListingMetadata listChildren(org.apache.hadoop.fs.Path path)
                                                         throws IOException

Description copied from interface: MetadataStore

Lists metadata for all direct children of a path.

Specified by:

listChildren in interface MetadataStore

Parameters:

path - the path to list

Returns:

metadata for all direct children of path which are being tracked by the MetadataStore, or null if the path was not found in the MetadataStore.

Throws:

IOException - if there is an error

addAncestors

@Retries.RetryTranslated
public void addAncestors(org.apache.hadoop.fs.Path qualifiedPath,
                                                  @Nullable
                                                  BulkOperationState operationState)
                                           throws IOException

This adds all new ancestors of a path as directories.

Important: to propagate TTL information, any new ancestors added must have their last updated timestamps set through S3Guard.patchLastUpdated(Collection, ITtlTimeProvider).

The implementation scans all up the directory tree and does a get() for each entry; at each level one is found it is added to the ancestor state.

The original implementation would stop on finding the first non-empty parent. This (re) implementation issues a GET for every parent entry and so detects and recovers from a tombstone marker further up the tree (i.e. an inconsistent store is corrected for).

if operationState is not null, when this method returns the operation state will be updated with all new entries created. This ensures that subsequent operations with the same store will not trigger new updates.

Specified by:

addAncestors in interface MetadataStore

Parameters:

qualifiedPath - path to update

operationState - (nullable) operational state for a bulk update

Throws:

IOException - on failure.

move

@Retries.RetryTranslated
public void move(@Nullable
                                          Collection<org.apache.hadoop.fs.Path> pathsToDelete,
                                          @Nullable
                                          Collection<PathMetadata> pathsToCreate,
                                          @Nullable
                                          BulkOperationState operationState)
                                   throws IOException

Record the effects of a FileSystem.rename(Path, Path) in the MetadataStore. Clients provide explicit enumeration of the affected paths (recursively), before and after the rename. This operation is not atomic, unless specific implementations claim otherwise. On the need to provide an enumeration of directory trees instead of just source and destination paths: Since a MetadataStore does not have to track all metadata for the underlying storage system, and a new MetadataStore may be created on an existing underlying filesystem, this move() may be the first time the MetadataStore sees the affected paths. Therefore, simply providing src and destination paths may not be enough to record the deletions (under src path) and creations (at destination) that are happening during the rename().. The DDB implementation sorts all the paths such that new items are ordered highest level entry first; deleted items are ordered lowest entry first. This is to ensure that if a client failed partway through the update, there will no entries in the table which lack parent entries.

Specified by:

move in interface MetadataStore

Parameters:

pathsToDelete - Collection of all paths that were removed from the source directory tree of the move.

pathsToCreate - Collection of all PathMetadata for the new paths that were created at the destination of the rename ().

operationState - Any ongoing state supplied to the rename tracker which is to be passed in with each move operation.

Throws:

IOException - if there is an error

put

@Retries.RetryTranslated
public void put(PathMetadata meta)
                                  throws IOException

Description copied from interface: MetadataStore

Saves metadata for exactly one path. Implementations may pre-create all the path's ancestors automatically. Implementations must update any DirListingMetadata objects which track the immediate parent of this file.

Specified by:

put in interface MetadataStore

Parameters:

meta - the metadata to save

Throws:

IOException - if there is an error

put

@Retries.RetryTranslated
public void put(PathMetadata meta,
                                         @Nullable
                                         BulkOperationState operationState)
                                  throws IOException

Description copied from interface: MetadataStore

Saves metadata for exactly one path, potentially using any bulk operation state to eliminate duplicate work. Implementations may pre-create all the path's ancestors automatically. Implementations must update any DirListingMetadata objects which track the immediate parent of this file.

Specified by:

put in interface MetadataStore

Parameters:

meta - the metadata to save

operationState - operational state for a bulk update

Throws:

IOException - if there is an error

put

@Retries.RetryTranslated
public void put(Collection<? extends PathMetadata> metas,
                                         @Nullable
                                         BulkOperationState operationState)
                                  throws IOException

Description copied from interface: MetadataStore

Saves metadata for any number of paths. Semantics are otherwise the same as single-path puts.

Specified by:

put in interface MetadataStore

Parameters:

metas - the metadata to save

operationState - (nullable) operational state for a bulk update

Throws:

IOException - if there is an error

put

@Retries.RetryTranslated
public void put(DirListingMetadata meta,
                                         List<org.apache.hadoop.fs.Path> unchangedEntries,
                                         @Nullable
                                         BulkOperationState operationState)
                                  throws IOException

Save directory listing metadata. Callers may save a partial directory listing for a given path, or may store a complete and authoritative copy of the directory listing. MetadataStore implementations may subsequently keep track of all modifications to the directory contents at this path, and return authoritative results from subsequent calls to MetadataStore.listChildren(Path). See DirListingMetadata. Any authoritative results returned are only authoritative for the scope of the MetadataStore: A per-process MetadataStore, for example, would only show results visible to that process, potentially missing metadata updates (create, delete) made to the same path by another process. To optimize updates and avoid overwriting existing entries which may contain extra data, entries in the list of unchangedEntries may be excluded. That is: the listing metadata has the full list of what it believes are children, but implementations can opt to ignore some.. There is retry around building the list of paths to update, but the call to processBatchWriteRequest(DynamoDBMetadataStore.AncestorState, PrimaryKey[], Item[]) is only tried once.

Specified by:

put in interface MetadataStore

Parameters:

meta - Directory listing metadata.

unchangedEntries - unchanged child entry paths

operationState - operational state for a bulk update

Throws:

IOException - IO problem

close

public void close()

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

destroy

@Retries.RetryTranslated
public void destroy()
                                      throws IOException

Description copied from interface: MetadataStore

Destroy all resources associated with the metadata store. The destroyed resources can be DynamoDB tables, MySQL databases/tables, or HDFS directories. Any operations after calling this method may possibly fail. This operation is idempotent.

Specified by:

destroy in interface MetadataStore

Throws:

IOException - if there is an error

prune

@Retries.RetryTranslated
public void prune(MetadataStore.PruneMode pruneMode,
                                           long cutoff)
                                    throws IOException

Description copied from interface: MetadataStore

Prune method with two modes of operation:

• MetadataStore.PruneMode.ALL_BY_MODTIME Clear any metadata older than a specified mod_time from the store. Note that this modification time is the S3 modification time from the object's metadata - from the object store. Implementations MUST clear file metadata, and MAY clear directory metadata (s3a itself does not track modification time for directories). Implementations may also choose to throw UnsupportedOperationException instead. Note that modification times must be in UTC, as returned by System.currentTimeMillis at the time of modification.

• MetadataStore.PruneMode.TOMBSTONES_BY_LASTUPDATED Clear any tombstone updated earlier than a specified time from the store. Note that this last_updated is the time when the metadata entry was last updated and maintained by the metadata store. Implementations MUST clear file metadata, and MAY clear directory metadata (s3a itself does not track modification time for directories). Implementations may also choose to throw UnsupportedOperationException instead. Note that last_updated must be in UTC, as returned by System.currentTimeMillis at the time of modification.

Specified by:

prune in interface MetadataStore

Parameters:

pruneMode - Prune Mode

cutoff - Oldest time to allow (UTC)

Throws:

IOException - if there is an error

prune

@Retries.RetryTranslated
public long prune(MetadataStore.PruneMode pruneMode,
                                           long cutoff,
                                           String keyPrefix)
                                    throws IOException

Prune files, in batches. There's optionally a sleep between each batch.

Specified by:

prune in interface MetadataStore

Parameters:

pruneMode - The mode of operation for the prune For details see MetadataStore.prune(PruneMode, long)

cutoff - Oldest modification time to allow

keyPrefix - The prefix for the keys that should be removed

Returns:

count of pruned items.

Throws:

IOException - Any IO/DDB failure.

InterruptedIOException - if the prune was interrupted

getAmazonDynamoDB

public com.amazonaws.services.dynamodbv2.AmazonDynamoDB getAmazonDynamoDB()

toString

public String toString()

Overrides:

toString in class Object

listAWSPolicyRules

public List<RoleModel.Statement> listAWSPolicyRules(Set<AWSPolicyProvider.AccessLevel> access)

The administrative policy includes all DDB table operations; application access is restricted to those operations S3Guard operations require when working with data in a guarded bucket.

Specified by:

listAWSPolicyRules in interface AWSPolicyProvider

Parameters:

access - access level desired.

Returns:

a possibly empty list of statements.

getTableName

public String getTableName()

getDiagnostics

@Retries.OnceRaw
public Map<String,String> getDiagnostics()
                                                   throws IOException

Description copied from interface: MetadataStore

Get any diagnostics information from a store, as a list of (key, value) tuples for display. Arbitrary values; no guarantee of stability. These are for debugging and testing only.

Specified by:

getDiagnostics in interface MetadataStore

Returns:

a map of strings.

Throws:

IOException - if there is an error

updateParameters

@Retries.OnceRaw
public void updateParameters(Map<String,String> parameters)
                                       throws IOException

Description copied from interface: MetadataStore

Tune/update parameters for an existing table.

Specified by:

updateParameters in interface MetadataStore

Parameters:

parameters - map of params to change.

Throws:

IOException - if there is an error

getReadThrottleEventCount

public long getReadThrottleEventCount()

Get the count of read throttle events.

Returns:

the current count of read throttle events.

getWriteThrottleEventCount

public long getWriteThrottleEventCount()

Get the count of write throttle events.

Returns:

the current count of write throttle events.

getScanThrottleEventCount

public long getScanThrottleEventCount()

Get the count of scan throttle events.

Returns:

the current count of scan throttle events.

getBatchWriteCapacityExceededCount

public long getBatchWriteCapacityExceededCount()

getInvoker

public Invoker getInvoker()

Get the operation invoker for write operations.

Returns:

an invoker for retrying mutating operations on a store.

wrapWithRetries

public <T> Iterable<T> wrapWithRetries(Iterable<T> source)

Wrap an iterator returned from any scan with a retrying one. This includes throttle handling. Retries will update the relevant counters/metrics for scan operations.

Parameters:

source - source iterator

Returns:

a retrying iterator.

initiateRenameOperation

public RenameTracker initiateRenameOperation(StoreContext storeContext,
                                             org.apache.hadoop.fs.Path source,
                                             S3AFileStatus sourceStatus,
                                             org.apache.hadoop.fs.Path dest)

Initiate the rename operation by creating the tracker for the filesystem to keep up to date with state changes in the S3A bucket.

Specified by:

initiateRenameOperation in interface MetadataStore

Parameters:

storeContext - store context.

source - source path

sourceStatus - status of the source file/dir

dest - destination path.

Returns:

the rename tracker

markAsAuthoritative

public int markAsAuthoritative(org.apache.hadoop.fs.Path dest,
                               BulkOperationState operationState)
                        throws IOException

Mark the directories instantiated under the destination path as authoritative. That is: all entries in the operationState (which must be an AncestorState instance), that are under the destination path. The database update synchronized on the operationState, so all other threads trying to update that state will be blocked until completion. This operation is only used in import and at the end of a rename, so this is not considered an issue.

Specified by:

markAsAuthoritative in interface MetadataStore

Parameters:

dest - destination path.

operationState - active state.

Returns:

the number of directories marked.

Throws:

IOException - failure.

initiateBulkWrite

public org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.AncestorState initiateBulkWrite(BulkOperationState.OperationType operation,
                                                                                              org.apache.hadoop.fs.Path dest)

Description copied from interface: MetadataStore

Initiate a bulk update and create an operation state for it. This may then be passed into put operations.

Specified by:

initiateBulkWrite in interface MetadataStore

Parameters:

operation - the type of the operation.

dest - path under which updates will be explicitly put.

Returns:

null or a store-specific state to pass into the put operations.

setTtlTimeProvider

public void setTtlTimeProvider(ITtlTimeProvider ttlTimeProvider)

Description copied from interface: MetadataStore

The TtlTimeProvider has to be set during the initialization for the metadatastore, but this method can be used for testing, and change the instance during runtime.

Specified by:

setTtlTimeProvider in interface MetadataStore

getInstrumentation

public MetastoreInstrumentation getInstrumentation()

Description copied from interface: MetadataStore

Get any S3GuardInstrumentation for this store...must not be null.

Specified by:

getInstrumentation in interface MetadataStore

Returns:

any store instrumentation.

getTableHandler

protected DynamoDBMetadataStoreTableManager getTableHandler()