org.apache.hadoop.fs.s3a.s3guard

Interface MetadataStore

All Superinterfaces:

AutoCloseable, Closeable

All Known Implementing Classes:

DynamoDBMetadataStore, LocalMetadataStore, NullMetadataStore

@InterfaceAudience.Private
 @InterfaceStability.Evolving
public interface MetadataStore
extends Closeable

MetadataStore defines the set of operations that any metadata store implementation must provide. Note that all Path objects provided to methods must be absolute, not relative paths. Implementations must implement any retries needed internally, such that transient errors are generally recovered from without throwing exceptions from this API.

Method Summary

Modifier and Type	Method and Description
void	delete(org.apache.hadoop.fs.Path path) Deletes exactly one path, leaving a tombstone to prevent lingering, inconsistent copies of it from being listed.
void	deleteSubtree(org.apache.hadoop.fs.Path path) Deletes the entire sub-tree rooted at the given path, leaving tombstones to prevent lingering, inconsistent copies of it from being listed.
void	destroy() Destroy all resources associated with the metadata store.
void	forgetMetadata(org.apache.hadoop.fs.Path path) Removes the record of exactly one path.
PathMetadata	get(org.apache.hadoop.fs.Path path) Gets metadata for a path.
PathMetadata	get(org.apache.hadoop.fs.Path path, boolean wantEmptyDirectoryFlag) Gets metadata for a path.
Map<String,String>	getDiagnostics() Get any diagnostics information from a store, as a list of (key, value) tuples for display.
void	initialize(org.apache.hadoop.conf.Configuration conf) Performs one-time initialization of the metadata store via configuration.
void	initialize(org.apache.hadoop.fs.FileSystem fs) Performs one-time initialization of the metadata store.
DirListingMetadata	listChildren(org.apache.hadoop.fs.Path path) Lists metadata for all direct children of a path.
void	move(Collection<org.apache.hadoop.fs.Path> pathsToDelete, Collection<PathMetadata> pathsToCreate) Record the effects of a FileSystem.rename(Path, Path) in the MetadataStore.
void	prune(long modTime) Clear any metadata older than a specified time from the repository.
void	prune(long modTime, String keyPrefix) Same as prune(long), but with an additional keyPrefix parameter to filter the pruned keys with a prefix.
void	put(Collection<PathMetadata> metas) Saves metadata for any number of paths.
void	put(DirListingMetadata meta) Save directory listing metadata.
void	put(PathMetadata meta) Saves metadata for exactly one path.
void	updateParameters(Map<String,String> parameters) Tune/update parameters for an existing table.

Methods inherited from interface java.io.Closeable

close

Method Detail

initialize

void initialize(org.apache.hadoop.fs.FileSystem fs)
         throws IOException

Performs one-time initialization of the metadata store.

Parameters:

fs - FileSystem associated with the MetadataStore

Throws:

IOException - if there is an error

initialize

void initialize(org.apache.hadoop.conf.Configuration conf)
         throws IOException

Performs one-time initialization of the metadata store via configuration.

Parameters:

conf - Configuration.

Throws:

IOException - if there is an error

See Also:

initialize(FileSystem)

delete

void delete(org.apache.hadoop.fs.Path path)
     throws IOException

Deletes exactly one path, leaving a tombstone to prevent lingering, inconsistent copies of it from being listed.

Parameters:

path - the path to delete

Throws:

IOException - if there is an error

forgetMetadata

void forgetMetadata(org.apache.hadoop.fs.Path path)
             throws IOException

Removes the record of exactly one path. Does not leave a tombstone (see delete(Path). It is currently intended for testing only, and a need to use it as part of normal FileSystem usage is not anticipated.

Parameters:

path - the path to delete

Throws:

IOException - if there is an error

deleteSubtree

void deleteSubtree(org.apache.hadoop.fs.Path path)
            throws IOException

Deletes the entire sub-tree rooted at the given path, leaving tombstones to prevent lingering, inconsistent copies of it from being listed. In addition to affecting future calls to get(Path), implementations must also update any stored DirListingMetadata objects which track the parent of this file.

Parameters:

path - the root of the sub-tree to delete

Throws:

IOException - if there is an error

get

PathMetadata get(org.apache.hadoop.fs.Path path)
          throws IOException

Gets metadata for a path.

Parameters:

path - the path to get

Returns:

metadata for path, null if not found

Throws:

IOException - if there is an error

get

PathMetadata get(org.apache.hadoop.fs.Path path,
                 boolean wantEmptyDirectoryFlag)
          throws IOException

Gets metadata for a path. Alternate method that includes a hint whether or not the MetadataStore should do work to compute the value for PathMetadata.isEmptyDirectory(). Since determining emptiness may be an expensive operation, this can save wasted work.

Parameters:

path - the path to get

wantEmptyDirectoryFlag - Set to true to give a hint to the MetadataStore that it should try to compute the empty directory flag.

Returns:

metadata for path, null if not found

Throws:

IOException - if there is an error

listChildren

DirListingMetadata listChildren(org.apache.hadoop.fs.Path path)
                         throws IOException

Lists metadata for all direct children of a path.

Parameters:

path - the path to list

Returns:

metadata for all direct children of path which are being tracked by the MetadataStore, or null if the path was not found in the MetadataStore.

Throws:

IOException - if there is an error

move

void move(Collection<org.apache.hadoop.fs.Path> pathsToDelete,
          Collection<PathMetadata> pathsToCreate)
   throws IOException

Record the effects of a FileSystem.rename(Path, Path) in the MetadataStore. Clients provide explicit enumeration of the affected paths (recursively), before and after the rename. This operation is not atomic, unless specific implementations claim otherwise. On the need to provide an enumeration of directory trees instead of just source and destination paths: Since a MetadataStore does not have to track all metadata for the underlying storage system, and a new MetadataStore may be created on an existing underlying filesystem, this move() may be the first time the MetadataStore sees the affected paths. Therefore, simply providing src and destination paths may not be enough to record the deletions (under src path) and creations (at destination) that are happening during the rename().

Parameters:

pathsToDelete - Collection of all paths that were removed from the source directory tree of the move.

pathsToCreate - Collection of all PathMetadata for the new paths that were created at the destination of the rename ().

Throws:

IOException - if there is an error

put

void put(PathMetadata meta)
  throws IOException

Saves metadata for exactly one path. Implementations may pre-create all the path's ancestors automatically. Implementations must update any DirListingMetadata objects which track the immediate parent of this file.

Parameters:

meta - the metadata to save

Throws:

IOException - if there is an error

put

void put(Collection<PathMetadata> metas)
  throws IOException

Saves metadata for any number of paths. Semantics are otherwise the same as single-path puts.

Parameters:

metas - the metadata to save

Throws:

IOException - if there is an error

put

void put(DirListingMetadata meta)
  throws IOException

Save directory listing metadata. Callers may save a partial directory listing for a given path, or may store a complete and authoritative copy of the directory listing. MetadataStore implementations may subsequently keep track of all modifications to the directory contents at this path, and return authoritative results from subsequent calls to listChildren(Path). See DirListingMetadata. Any authoritative results returned are only authoritative for the scope of the MetadataStore: A per-process MetadataStore, for example, would only show results visible to that process, potentially missing metadata updates (create, delete) made to the same path by another process.

Parameters:

meta - Directory listing metadata.

Throws:

IOException - if there is an error

destroy

void destroy()
      throws IOException

Destroy all resources associated with the metadata store. The destroyed resources can be DynamoDB tables, MySQL databases/tables, or HDFS directories. Any operations after calling this method may possibly fail. This operation is idempotent.

Throws:

IOException - if there is an error

prune

void prune(long modTime)
    throws IOException,
           UnsupportedOperationException

Clear any metadata older than a specified time from the repository. Implementations MUST clear file metadata, and MAY clear directory metadata (s3a itself does not track modification time for directories). Implementations may also choose to throw UnsupportedOperationException istead. Note that modification times should be in UTC, as returned by System.currentTimeMillis at the time of modification.

Parameters:

modTime - Oldest modification time to allow

Throws:

IOException - if there is an error

UnsupportedOperationException - if not implemented

prune

void prune(long modTime,
           String keyPrefix)
    throws IOException,
           UnsupportedOperationException

Same as prune(long), but with an additional keyPrefix parameter to filter the pruned keys with a prefix.

Parameters:

modTime - Oldest modification time to allow

keyPrefix - The prefix for the keys that should be removed

Throws:

IOException - if there is an error

UnsupportedOperationException - if not implemented

getDiagnostics

Map<String,String> getDiagnostics()
                           throws IOException

Get any diagnostics information from a store, as a list of (key, value) tuples for display. Arbitrary values; no guarantee of stability. These are for debugging and testing only.

Returns:

a map of strings.

Throws:

IOException - if there is an error

updateParameters

void updateParameters(Map<String,String> parameters)
               throws IOException

Tune/update parameters for an existing table.

Parameters:

parameters - map of params to change.

Throws:

IOException - if there is an error