DynamoDBMetadataStore (Apache Hadoop Amazon Web Services support 3.2.0 API)

java.lang.Object
- org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore

All Implemented Interfaces:

Closeable, AutoCloseable, MetadataStore
```
@InterfaceAudience.Private
 @InterfaceStability.Evolving
public class DynamoDBMetadataStore
extends Object
implements MetadataStore
```
DynamoDBMetadataStore is a MetadataStore that persists file system metadata to DynamoDB. The current implementation uses a schema consisting of a single table. The name of the table can be configured by config key Constants.S3GUARD_DDB_TABLE_NAME_KEY. By default, it matches the name of the S3 bucket. Each item in the table represents a single directory or file. Its path is split into separate table attributes:
- parent (absolute path of the parent, with bucket name inserted as first path component).
- child (path of that specific child, relative to parent).
- optional boolean attribute tracking whether the path is a directory. Absence or a false value indicates the path is a file.
- optional long attribute revealing modification time of file. This attribute is meaningful only to file items.
- optional long attribute revealing file length. This attribute is meaningful only to file items.
- optional long attribute revealing block size of the file. This attribute is meaningful only to file items.
The DynamoDB partition key is the parent, and the range key is the child. To allow multiple buckets to share the same DynamoDB table, the bucket name is treated as the root directory. For example, assume the consistent store contains metadata representing this file system structure:
```
 s3a://bucket/dir1
 |-- dir2
 |   |-- file1
 |   `-- file2
 `-- dir3
     |-- dir4
     |   `-- file3
     |-- dir5
     |   `-- file4
     `-- dir6
 
```
This is persisted to a single DynamoDB table as:
```
 =========================================================================
 | parent                 | child | is_dir | mod_time | len |     ...    |
 =========================================================================
 | /bucket                | dir1  | true   |          |     |            |
 | /bucket/dir1           | dir2  | true   |          |     |            |
 | /bucket/dir1           | dir3  | true   |          |     |            |
 | /bucket/dir1/dir2      | file1 |        |   100    | 111 |            |
 | /bucket/dir1/dir2      | file2 |        |   200    | 222 |            |
 | /bucket/dir1/dir3      | dir4  | true   |          |     |            |
 | /bucket/dir1/dir3      | dir5  | true   |          |     |            |
 | /bucket/dir1/dir3/dir4 | file3 |        |   300    | 333 |            |
 | /bucket/dir1/dir3/dir5 | file4 |        |   400    | 444 |            |
 | /bucket/dir1/dir3      | dir6  | true   |          |     |            |
 =========================================================================
 
```
This choice of schema is efficient for read access patterns. get(Path) can be served from a single item lookup. listChildren(Path) can be served from a query against all rows matching the parent (the partition key) and the returned list is guaranteed to be sorted by child (the range key). Tracking whether or not a path is a directory helps prevent unnecessary queries during traversal of an entire sub-tree. Some mutating operations, notably deleteSubtree(Path) and move(Collection, Collection), are less efficient with this schema. They require mutating multiple items in the DynamoDB table. By default, DynamoDB access is performed within the same AWS region as the S3 bucket that hosts the S3A instance. During initialization, it checks the location of the S3 bucket and creates a DynamoDB client connected to the same region. The region may also be set explicitly by setting the config parameter fs.s3a.s3guard.ddb.region to the corresponding region.

Field Summary

Fields
Modifier and Type	Field and Description
`static String`	`E_INCOMPATIBLE_VERSION` Error: version mismatch.
`static String`	`E_NO_VERSION_MARKER` Error: version marker not found in table.
`static org.slf4j.Logger`	`LOG`
`static int`	`VERSION` Current version number.
`static String`	`VERSION_MARKER` parent/child name to use in the version marker.

Constructor Summary

Constructors
Constructor and Description

DynamoDBMetadataStore()

Constructors
Constructor and Description
`DynamoDBMetadataStore()`

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`close()`
`void`	`delete(org.apache.hadoop.fs.Path path)` Deletes exactly one path, leaving a tombstone to prevent lingering, inconsistent copies of it from being listed.
`void`	`deleteSubtree(org.apache.hadoop.fs.Path path)` Deletes the entire sub-tree rooted at the given path, leaving tombstones to prevent lingering, inconsistent copies of it from being listed.
`void`	`destroy()` Destroy all resources associated with the metadata store.
`static com.amazonaws.SdkBaseException`	`extractInnerException(IllegalArgumentException ex)` Take an `IllegalArgumentException` raised by a DDB operation and if it contains an inner SDK exception, unwrap it.
`void`	`forgetMetadata(org.apache.hadoop.fs.Path path)` Removes the record of exactly one path.
`DDBPathMetadata`	`get(org.apache.hadoop.fs.Path path)` Gets metadata for a path.
`DDBPathMetadata`	`get(org.apache.hadoop.fs.Path path, boolean wantEmptyDirectoryFlag)` Gets metadata for a path.
`com.amazonaws.services.dynamodbv2.AmazonDynamoDB`	`getAmazonDynamoDB()`
`long`	`getBatchWriteCapacityExceededCount()`
`Map<String,String>`	`getDiagnostics()` Get any diagnostics information from a store, as a list of (key, value) tuples for display.
`Invoker`	`getInvoker()`
`long`	`getReadThrottleEventCount()` Get the count of read throttle events.
`String`	`getTableName()`
`long`	`getWriteThrottleEventCount()` Get the count of write throttle events.
`void`	`initialize(org.apache.hadoop.conf.Configuration config)` Performs one-time initialization of the metadata store via configuration.
`void`	`initialize(org.apache.hadoop.fs.FileSystem fs)` Performs one-time initialization of the metadata store.
`DirListingMetadata`	`listChildren(org.apache.hadoop.fs.Path path)` Lists metadata for all direct children of a path.
`void`	`move(Collection<org.apache.hadoop.fs.Path> pathsToDelete, Collection<PathMetadata> pathsToCreate)` Record the effects of a `FileSystem.rename(Path, Path)` in the MetadataStore.
`void`	`prune(long modTime)` Clear any metadata older than a specified time from the repository.
`void`	`prune(long modTime, String keyPrefix)` Prune files, in batches.
`void`	`put(Collection<PathMetadata> metas)` Saves metadata for any number of paths.
`void`	`put(DirListingMetadata meta)` Save directory listing metadata.
`void`	`put(PathMetadata meta)` Saves metadata for exactly one path.
`void`	`tagTable()` Add tags from configuration to the existing DynamoDB table.
`String`	`toString()`
`void`	`updateParameters(Map<String,String> parameters)` Tune/update parameters for an existing table.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

- Field Detail
  - LOG
```
public static final org.slf4j.Logger LOG
```
  - VERSION_MARKER
```
public static final String VERSION_MARKER
```
    parent/child name to use in the version marker.
    
    See Also:
    
    Constant Field Values
  - VERSION
```
public static final int VERSION
```
    Current version number.
    
    See Also:
    
    Constant Field Values
  - E_NO_VERSION_MARKER
```
public static final String E_NO_VERSION_MARKER
```
    Error: version marker not found in table.
    
    See Also:
    
    Constant Field Values
  - E_INCOMPATIBLE_VERSION
```
public static final String E_INCOMPATIBLE_VERSION
```
    Error: version mismatch.
    
    See Also:
    
    Constant Field Values
- Constructor Detail
  - DynamoDBMetadataStore
```
public DynamoDBMetadataStore()
```
- Method Detail
  - initialize
```
@Retries.OnceRaw
public void initialize(org.apache.hadoop.fs.FileSystem fs)
                                 throws IOException
```
    Performs one-time initialization of the metadata store.. The credentials for authenticating with S3 are requested from the FS via S3AFileSystem.shareCredentials(String); this will increment the reference counter of these credentials.
    
    Specified by:
    
    initialize in interface MetadataStore
    
    Parameters:
    
    fs - S3AFileSystem associated with the MetadataStore
    
    Throws:
    
    IOException - on a failure
  - initialize
```
@Retries.OnceRaw
public void initialize(org.apache.hadoop.conf.Configuration config)
                                 throws IOException
```
    Performs one-time initialization of the metadata store via configuration. This initialization depends on the configuration object to get AWS credentials, DynamoDBFactory implementation class, DynamoDB endpoints, DynamoDB table names etc. After initialization, this metadata store does not explicitly relate to any S3 bucket, which be nonexistent. This is used to operate the metadata store directly beyond the scope of the S3AFileSystem integration, e.g. command line tools. Generally, callers should use initialize(FileSystem) with an initialized S3AFileSystem instance. Without a filesystem to act as a reference point, the configuration itself must declare the table name and region in the Constants.S3GUARD_DDB_TABLE_NAME_KEY and Constants.S3GUARD_DDB_REGION_KEY respectively. It also creates a new credential provider list from the configuration, using the base fs.s3a.* options, as there is no bucket to infer per-bucket settings from.
    
    Specified by:
    
    initialize in interface MetadataStore
    
    Parameters:
    
    config - Configuration.
    
    Throws:
    
    IOException - if there is an error
    
    IllegalArgumentException - if the configuration is incomplete
    
    See Also:
    
    initialize(FileSystem)
  - delete
```
@Retries.RetryTranslated
public void delete(org.apache.hadoop.fs.Path path)
                                     throws IOException
```
    Description copied from interface: MetadataStore
    
    Deletes exactly one path, leaving a tombstone to prevent lingering, inconsistent copies of it from being listed.
    
    Specified by:
    
    delete in interface MetadataStore
    
    Parameters:
    
    path - the path to delete
    
    Throws:
    
    IOException - if there is an error
  - forgetMetadata
```
@Retries.RetryTranslated
public void forgetMetadata(org.apache.hadoop.fs.Path path)
                                             throws IOException
```
    Description copied from interface: MetadataStore
    
    Removes the record of exactly one path. Does not leave a tombstone (see MetadataStore.delete(Path). It is currently intended for testing only, and a need to use it as part of normal FileSystem usage is not anticipated.
    
    Specified by:
    
    forgetMetadata in interface MetadataStore
    
    Parameters:
    
    path - the path to delete
    
    Throws:
    
    IOException - if there is an error
  - deleteSubtree
```
@Retries.RetryTranslated
public void deleteSubtree(org.apache.hadoop.fs.Path path)
                                            throws IOException
```
    Description copied from interface: MetadataStore
    
    Deletes the entire sub-tree rooted at the given path, leaving tombstones to prevent lingering, inconsistent copies of it from being listed. In addition to affecting future calls to MetadataStore.get(Path), implementations must also update any stored DirListingMetadata objects which track the parent of this file.
    
    Specified by:
    
    deleteSubtree in interface MetadataStore
    
    Parameters:
    
    path - the root of the sub-tree to delete
    
    Throws:
    
    IOException - if there is an error
  - get
```
@Retries.RetryTranslated
public DDBPathMetadata get(org.apache.hadoop.fs.Path path)
                                             throws IOException
```
    Description copied from interface: MetadataStore
    
    Gets metadata for a path.
    
    Specified by:
    
    get in interface MetadataStore
    
    Parameters:
    
    path - the path to get
    
    Returns:
    
    metadata for path, null if not found
    
    Throws:
    
    IOException - if there is an error
  - get
```
@Retries.RetryTranslated
public DDBPathMetadata get(org.apache.hadoop.fs.Path path,
                                                    boolean wantEmptyDirectoryFlag)
                                             throws IOException
```
    Description copied from interface: MetadataStore
    
    Gets metadata for a path. Alternate method that includes a hint whether or not the MetadataStore should do work to compute the value for PathMetadata.isEmptyDirectory(). Since determining emptiness may be an expensive operation, this can save wasted work.
    
    Specified by:
    
    get in interface MetadataStore
    
    Parameters:
    
    path - the path to get
    
    wantEmptyDirectoryFlag - Set to true to give a hint to the MetadataStore that it should try to compute the empty directory flag.
    
    Returns:
    
    metadata for path, null if not found
    
    Throws:
    
    IOException - if there is an error
  - listChildren
```
@Retries.RetryTranslated
public DirListingMetadata listChildren(org.apache.hadoop.fs.Path path)
                                                         throws IOException
```
    Description copied from interface: MetadataStore
    
    Lists metadata for all direct children of a path.
    
    Specified by:
    
    listChildren in interface MetadataStore
    
    Parameters:
    
    path - the path to list
    
    Returns:
    
    metadata for all direct children of path which are being tracked by the MetadataStore, or null if the path was not found in the MetadataStore.
    
    Throws:
    
    IOException - if there is an error
  - move
```
@Retries.RetryTranslated
public void move(Collection<org.apache.hadoop.fs.Path> pathsToDelete,
                                          Collection<PathMetadata> pathsToCreate)
                                   throws IOException
```
    Description copied from interface: MetadataStore
    
    Record the effects of a FileSystem.rename(Path, Path) in the MetadataStore. Clients provide explicit enumeration of the affected paths (recursively), before and after the rename. This operation is not atomic, unless specific implementations claim otherwise. On the need to provide an enumeration of directory trees instead of just source and destination paths: Since a MetadataStore does not have to track all metadata for the underlying storage system, and a new MetadataStore may be created on an existing underlying filesystem, this move() may be the first time the MetadataStore sees the affected paths. Therefore, simply providing src and destination paths may not be enough to record the deletions (under src path) and creations (at destination) that are happening during the rename().
    
    Specified by:
    
    move in interface MetadataStore
    
    Parameters:
    
    pathsToDelete - Collection of all paths that were removed from the source directory tree of the move.
    
    pathsToCreate - Collection of all PathMetadata for the new paths that were created at the destination of the rename ().
    
    Throws:
    
    IOException - if there is an error
  - put
```
@Retries.RetryTranslated
public void put(PathMetadata meta)
                                  throws IOException
```
    Description copied from interface: MetadataStore
    
    Saves metadata for exactly one path. Implementations may pre-create all the path's ancestors automatically. Implementations must update any DirListingMetadata objects which track the immediate parent of this file.
    
    Specified by:
    
    put in interface MetadataStore
    
    Parameters:
    
    meta - the metadata to save
    
    Throws:
    
    IOException - if there is an error
  - put
```
@Retries.RetryTranslated
public void put(Collection<PathMetadata> metas)
                                  throws IOException
```
    Description copied from interface: MetadataStore
    
    Saves metadata for any number of paths. Semantics are otherwise the same as single-path puts.
    
    Specified by:
    
    put in interface MetadataStore
    
    Parameters:
    
    metas - the metadata to save
    
    Throws:
    
    IOException - if there is an error
  - put
```
@Retries.RetryTranslated
public void put(DirListingMetadata meta)
                                  throws IOException
```
    Save directory listing metadata. Callers may save a partial directory listing for a given path, or may store a complete and authoritative copy of the directory listing. MetadataStore implementations may subsequently keep track of all modifications to the directory contents at this path, and return authoritative results from subsequent calls to MetadataStore.listChildren(Path). See DirListingMetadata. Any authoritative results returned are only authoritative for the scope of the MetadataStore: A per-process MetadataStore, for example, would only show results visible to that process, potentially missing metadata updates (create, delete) made to the same path by another process.. There is retry around building the list of paths to update, but the call to processBatchWriteRequest(PrimaryKey[], Item[]) is only tried once.
    
    Specified by:
    
    put in interface MetadataStore
    
    Parameters:
    
    meta - Directory listing metadata.
    
    Throws:
    
    IOException - IO problem
  - close
```
public void close()
```
    Specified by:
    
    close in interface Closeable
    
    Specified by:
    
    close in interface AutoCloseable
  - destroy
```
@Retries.RetryTranslated
public void destroy()
                                      throws IOException
```
    Description copied from interface: MetadataStore
    
    Destroy all resources associated with the metadata store. The destroyed resources can be DynamoDB tables, MySQL databases/tables, or HDFS directories. Any operations after calling this method may possibly fail. This operation is idempotent.
    
    Specified by:
    
    destroy in interface MetadataStore
    
    Throws:
    
    IOException - if there is an error
  - prune
```
@Retries.RetryTranslated
public void prune(long modTime)
                                    throws IOException
```
    Description copied from interface: MetadataStore
    
    Clear any metadata older than a specified time from the repository. Implementations MUST clear file metadata, and MAY clear directory metadata (s3a itself does not track modification time for directories). Implementations may also choose to throw UnsupportedOperationException istead. Note that modification times should be in UTC, as returned by System.currentTimeMillis at the time of modification.
    
    Specified by:
    
    prune in interface MetadataStore
    
    Parameters:
    
    modTime - Oldest modification time to allow
    
    Throws:
    
    IOException - if there is an error
  - prune
```
@Retries.RetryTranslated
public void prune(long modTime,
                                           String keyPrefix)
                                    throws IOException
```
    Prune files, in batches. There's a sleep between each batch.
    
    Specified by:
    
    prune in interface MetadataStore
    
    Parameters:
    
    modTime - Oldest modification time to allow
    
    keyPrefix - The prefix for the keys that should be removed
    
    Throws:
    
    IOException - Any IO/DDB failure.
    
    InterruptedIOException - if the prune was interrupted
  - tagTable
```
@Retries.OnceRaw
public void tagTable()
```
    Add tags from configuration to the existing DynamoDB table.
  - getAmazonDynamoDB
```
public com.amazonaws.services.dynamodbv2.AmazonDynamoDB getAmazonDynamoDB()
```
  - toString
```
public String toString()
```
    Overrides:
    
    toString in class Object
  - getTableName
```
public String getTableName()
```
  - getDiagnostics
```
@Retries.OnceRaw
public Map<String,String> getDiagnostics()
                                                   throws IOException
```
    Description copied from interface: MetadataStore
    
    Get any diagnostics information from a store, as a list of (key, value) tuples for display. Arbitrary values; no guarantee of stability. These are for debugging and testing only.
    
    Specified by:
    
    getDiagnostics in interface MetadataStore
    
    Returns:
    
    a map of strings.
    
    Throws:
    
    IOException - if there is an error
  - updateParameters
```
@Retries.OnceRaw
public void updateParameters(Map<String,String> parameters)
                                       throws IOException
```
    Description copied from interface: MetadataStore
    
    Tune/update parameters for an existing table.
    
    Specified by:
    
    updateParameters in interface MetadataStore
    
    Parameters:
    
    parameters - map of params to change.
    
    Throws:
    
    IOException - if there is an error
  - getReadThrottleEventCount
```
public long getReadThrottleEventCount()
```
    Get the count of read throttle events.
    
    Returns:
    
    the current count of read throttle events.
  - getWriteThrottleEventCount
```
public long getWriteThrottleEventCount()
```
    Get the count of write throttle events.
    
    Returns:
    
    the current count of write throttle events.
  - getBatchWriteCapacityExceededCount
```
public long getBatchWriteCapacityExceededCount()
```
  - getInvoker
```
public Invoker getInvoker()
```
  - extractInnerException
```
public static com.amazonaws.SdkBaseException extractInnerException(IllegalArgumentException ex)
```
    Take an IllegalArgumentException raised by a DDB operation and if it contains an inner SDK exception, unwrap it.
    
    Parameters:
    
    ex - exception.
    
    Returns:
    
    the inner AWS exception or null.

Class DynamoDBMetadataStore

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

LOG

VERSION_MARKER

VERSION

E_NO_VERSION_MARKER

E_INCOMPATIBLE_VERSION

Constructor Detail

DynamoDBMetadataStore

Method Detail

initialize

initialize

delete

forgetMetadata

deleteSubtree

get

get

listChildren

move

put

put

put

close

destroy

prune

prune

tagTable

getAmazonDynamoDB

toString

getTableName

getDiagnostics

updateParameters

getReadThrottleEventCount

getWriteThrottleEventCount

getBatchWriteCapacityExceededCount

getInvoker

extractInnerException