org.apache.hadoop.fs.s3a.s3guard

Class NullMetadataStore

java.lang.Object

org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore

All Implemented Interfaces:

Closeable, AutoCloseable, MetadataStore

public class NullMetadataStore
extends Object
implements MetadataStore

A no-op implementation of MetadataStore. Clients that use this implementation should behave the same as they would without any MetadataStore.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.fs.s3a.s3guard.MetadataStore

MetadataStore.PruneMode

Constructor Summary

Constructors

Constructor and Description
NullMetadataStore()

Method Summary

Modifier and Type	Method and Description
void	addAncestors(org.apache.hadoop.fs.Path qualifiedPath, BulkOperationState operationState) This adds all new ancestors of a path as directories.
void	close()
void	delete(org.apache.hadoop.fs.Path path, BulkOperationState operationState) Deletes exactly one path, leaving a tombstone to prevent lingering, inconsistent copies of it from being listed.
void	deletePaths(Collection<org.apache.hadoop.fs.Path> paths, BulkOperationState operationState) Delete the paths.
void	deleteSubtree(org.apache.hadoop.fs.Path path, BulkOperationState operationState) Deletes the entire sub-tree rooted at the given path, leaving tombstones to prevent lingering, inconsistent copies of it from being listed.
void	destroy() Destroy all resources associated with the metadata store.
void	forgetMetadata(org.apache.hadoop.fs.Path path) Removes the record of exactly one path.
PathMetadata	get(org.apache.hadoop.fs.Path path) Gets metadata for a path.
PathMetadata	get(org.apache.hadoop.fs.Path path, boolean wantEmptyDirectoryFlag) Gets metadata for a path.
Map<String,String>	getDiagnostics() Get any diagnostics information from a store, as a list of (key, value) tuples for display.
void	initialize(org.apache.hadoop.conf.Configuration conf, ITtlTimeProvider ttlTimeProvider) Performs one-time initialization of the metadata store via configuration.
void	initialize(org.apache.hadoop.fs.FileSystem fs, ITtlTimeProvider ttlTimeProvider) Performs one-time initialization of the metadata store.
RenameTracker	initiateRenameOperation(StoreContext storeContext, org.apache.hadoop.fs.Path source, S3AFileStatus sourceStatus, org.apache.hadoop.fs.Path dest) Start a rename operation.
DirListingMetadata	listChildren(org.apache.hadoop.fs.Path path) Lists metadata for all direct children of a path.
void	move(Collection<org.apache.hadoop.fs.Path> pathsToDelete, Collection<PathMetadata> pathsToCreate, BulkOperationState operationState) Record the effects of a FileSystem.rename(Path, Path) in the MetadataStore.
void	prune(MetadataStore.PruneMode pruneMode, long cutoff) Prune method with two modes of operation: MetadataStore.PruneMode.ALL_BY_MODTIME Clear any metadata older than a specified mod_time from the store.
long	prune(MetadataStore.PruneMode pruneMode, long cutoff, String keyPrefix) Same as MetadataStore.prune(PruneMode, long), but with an additional keyPrefix parameter to filter the pruned keys with a prefix.
void	put(Collection<? extends PathMetadata> meta, BulkOperationState operationState) Saves metadata for any number of paths.
void	put(DirListingMetadata meta, List<org.apache.hadoop.fs.Path> unchangedEntries, BulkOperationState operationState) Save directory listing metadata.
void	put(PathMetadata meta) Saves metadata for exactly one path.
void	put(PathMetadata meta, BulkOperationState operationState) Saves metadata for exactly one path, potentially using any bulk operation state to eliminate duplicate work.
void	setTtlTimeProvider(ITtlTimeProvider ttlTimeProvider) The TtlTimeProvider has to be set during the initialization for the metadatastore, but this method can be used for testing, and change the instance during runtime.
String	toString()
void	updateParameters(Map<String,String> parameters) Tune/update parameters for an existing table.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.fs.s3a.s3guard.MetadataStore

getInstrumentation, initiateBulkWrite, markAsAuthoritative

Constructor Detail

NullMetadataStore

public NullMetadataStore()

Method Detail

initialize

public void initialize(org.apache.hadoop.fs.FileSystem fs,
                       ITtlTimeProvider ttlTimeProvider)
                throws IOException

Description copied from interface: MetadataStore

Performs one-time initialization of the metadata store.

Specified by:

initialize in interface MetadataStore

Parameters:

fs - FileSystem associated with the MetadataStore

ttlTimeProvider - the time provider to use for metadata expiry

Throws:

IOException - if there is an error

initialize

public void initialize(org.apache.hadoop.conf.Configuration conf,
                       ITtlTimeProvider ttlTimeProvider)
                throws IOException

Description copied from interface: MetadataStore

Performs one-time initialization of the metadata store via configuration.

Specified by:

initialize in interface MetadataStore

Parameters:

conf - Configuration.

ttlTimeProvider - the time provider to use for metadata expiry

Throws:

IOException - if there is an error

See Also:

MetadataStore.initialize(FileSystem, ITtlTimeProvider)

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

delete

public void delete(org.apache.hadoop.fs.Path path,
                   BulkOperationState operationState)
            throws IOException

Description copied from interface: MetadataStore

Deletes exactly one path, leaving a tombstone to prevent lingering, inconsistent copies of it from being listed. Deleting an entry with a tombstone needs a S3Guard.TtlTimeProvider because the lastUpdated field of the record has to be updated to

now

.

Specified by:

delete in interface MetadataStore

Parameters:

path - the path to delete

operationState - (nullable) operational state for a bulk update

Throws:

IOException - if there is an error

forgetMetadata

public void forgetMetadata(org.apache.hadoop.fs.Path path)
                    throws IOException

Description copied from interface: MetadataStore

Removes the record of exactly one path. Does not leave a tombstone (see MetadataStore.delete(Path, BulkOperationState). It is currently intended for testing only, and a need to use it as part of normal FileSystem usage is not anticipated.

Specified by:

forgetMetadata in interface MetadataStore

Parameters:

path - the path to delete

Throws:

IOException - if there is an error

deleteSubtree

public void deleteSubtree(org.apache.hadoop.fs.Path path,
                          BulkOperationState operationState)
                   throws IOException

Description copied from interface: MetadataStore

Deletes the entire sub-tree rooted at the given path, leaving tombstones to prevent lingering, inconsistent copies of it from being listed. In addition to affecting future calls to MetadataStore.get(Path), implementations must also update any stored DirListingMetadata objects which track the parent of this file. Deleting a subtree with a tombstone needs a S3Guard.TtlTimeProvider because the lastUpdated field of all records have to be updated to

now

.

Specified by:

deleteSubtree in interface MetadataStore

Parameters:

path - the root of the sub-tree to delete

operationState - (nullable) operational state for a bulk update

Throws:

IOException - if there is an error

deletePaths

public void deletePaths(Collection<org.apache.hadoop.fs.Path> paths,
                        @Nullable
                        BulkOperationState operationState)
                 throws IOException

Description copied from interface: MetadataStore

Delete the paths. There's no attempt to order the paths: they are deleted in the order passed in.

Specified by:

deletePaths in interface MetadataStore

Parameters:

paths - paths to delete.

operationState - Nullable operation state

Throws:

IOException - failure

get

public PathMetadata get(org.apache.hadoop.fs.Path path)
                 throws IOException

Description copied from interface: MetadataStore

Gets metadata for a path.

Specified by:

get in interface MetadataStore

Parameters:

path - the path to get

Returns:

metadata for path, null if not found

Throws:

IOException - if there is an error

get

public PathMetadata get(org.apache.hadoop.fs.Path path,
                        boolean wantEmptyDirectoryFlag)
                 throws IOException

Description copied from interface: MetadataStore

Gets metadata for a path. Alternate method that includes a hint whether or not the MetadataStore should do work to compute the value for PathMetadata.isEmptyDirectory(). Since determining emptiness may be an expensive operation, this can save wasted work.

Specified by:

get in interface MetadataStore

Parameters:

path - the path to get

wantEmptyDirectoryFlag - Set to true to give a hint to the MetadataStore that it should try to compute the empty directory flag.

Returns:

metadata for path, null if not found

Throws:

IOException - if there is an error

listChildren

public DirListingMetadata listChildren(org.apache.hadoop.fs.Path path)
                                throws IOException

Description copied from interface: MetadataStore

Lists metadata for all direct children of a path.

Specified by:

listChildren in interface MetadataStore

Parameters:

path - the path to list

Returns:

metadata for all direct children of path which are being tracked by the MetadataStore, or null if the path was not found in the MetadataStore.

Throws:

IOException - if there is an error

move

public void move(Collection<org.apache.hadoop.fs.Path> pathsToDelete,
                 Collection<PathMetadata> pathsToCreate,
                 BulkOperationState operationState)
          throws IOException

Description copied from interface: MetadataStore

Record the effects of a FileSystem.rename(Path, Path) in the MetadataStore. Clients provide explicit enumeration of the affected paths (recursively), before and after the rename. This operation is not atomic, unless specific implementations claim otherwise. On the need to provide an enumeration of directory trees instead of just source and destination paths: Since a MetadataStore does not have to track all metadata for the underlying storage system, and a new MetadataStore may be created on an existing underlying filesystem, this move() may be the first time the MetadataStore sees the affected paths. Therefore, simply providing src and destination paths may not be enough to record the deletions (under src path) and creations (at destination) that are happening during the rename().

Specified by:

move in interface MetadataStore

Parameters:

pathsToDelete - Collection of all paths that were removed from the source directory tree of the move.

pathsToCreate - Collection of all PathMetadata for the new paths that were created at the destination of the rename().

operationState - Any ongoing state supplied to the rename tracker which is to be passed in with each move operation.

Throws:

IOException - if there is an error

put

public void put(PathMetadata meta)
         throws IOException

Description copied from interface: MetadataStore

Saves metadata for exactly one path. Implementations may pre-create all the path's ancestors automatically. Implementations must update any DirListingMetadata objects which track the immediate parent of this file.

Specified by:

put in interface MetadataStore

Parameters:

meta - the metadata to save

Throws:

IOException - if there is an error

put

public void put(PathMetadata meta,
                BulkOperationState operationState)
         throws IOException

Description copied from interface: MetadataStore

Saves metadata for exactly one path, potentially using any bulk operation state to eliminate duplicate work. Implementations may pre-create all the path's ancestors automatically. Implementations must update any DirListingMetadata objects which track the immediate parent of this file.

Specified by:

put in interface MetadataStore

Parameters:

meta - the metadata to save

operationState - operational state for a bulk update

Throws:

IOException - if there is an error

put

public void put(Collection<? extends PathMetadata> meta,
                BulkOperationState operationState)
         throws IOException

Description copied from interface: MetadataStore

Saves metadata for any number of paths. Semantics are otherwise the same as single-path puts.

Specified by:

put in interface MetadataStore

Parameters:

meta - the metadata to save

operationState - (nullable) operational state for a bulk update

Throws:

IOException - if there is an error

put

public void put(DirListingMetadata meta,
                List<org.apache.hadoop.fs.Path> unchangedEntries,
                BulkOperationState operationState)
         throws IOException

Description copied from interface: MetadataStore

Save directory listing metadata. Callers may save a partial directory listing for a given path, or may store a complete and authoritative copy of the directory listing. MetadataStore implementations may subsequently keep track of all modifications to the directory contents at this path, and return authoritative results from subsequent calls to MetadataStore.listChildren(Path). See DirListingMetadata. Any authoritative results returned are only authoritative for the scope of the MetadataStore: A per-process MetadataStore, for example, would only show results visible to that process, potentially missing metadata updates (create, delete) made to the same path by another process. To optimize updates and avoid overwriting existing entries which may contain extra data, entries in the list of unchangedEntries may be excluded. That is: the listing metadata has the full list of what it believes are children, but implementations can opt to ignore some.

Specified by:

put in interface MetadataStore

Parameters:

meta - Directory listing metadata.

unchangedEntries - list of entries in the dir listing which have not changed since the directory was list scanned on s3guard.

operationState - operational state for a bulk update

Throws:

IOException - if there is an error

destroy

public void destroy()
             throws IOException

Description copied from interface: MetadataStore

Destroy all resources associated with the metadata store. The destroyed resources can be DynamoDB tables, MySQL databases/tables, or HDFS directories. Any operations after calling this method may possibly fail. This operation is idempotent.

Specified by:

destroy in interface MetadataStore

Throws:

IOException - if there is an error

prune

public void prune(MetadataStore.PruneMode pruneMode,
                  long cutoff)

Description copied from interface: MetadataStore

Prune method with two modes of operation:

• MetadataStore.PruneMode.ALL_BY_MODTIME Clear any metadata older than a specified mod_time from the store. Note that this modification time is the S3 modification time from the object's metadata - from the object store. Implementations MUST clear file metadata, and MAY clear directory metadata (s3a itself does not track modification time for directories). Implementations may also choose to throw UnsupportedOperationException instead. Note that modification times must be in UTC, as returned by System.currentTimeMillis at the time of modification.

• MetadataStore.PruneMode.TOMBSTONES_BY_LASTUPDATED Clear any tombstone updated earlier than a specified time from the store. Note that this last_updated is the time when the metadata entry was last updated and maintained by the metadata store. Implementations MUST clear file metadata, and MAY clear directory metadata (s3a itself does not track modification time for directories). Implementations may also choose to throw UnsupportedOperationException instead. Note that last_updated must be in UTC, as returned by System.currentTimeMillis at the time of modification.

Specified by:

prune in interface MetadataStore

Parameters:

pruneMode - Prune Mode

cutoff - Oldest time to allow (UTC)

prune

public long prune(MetadataStore.PruneMode pruneMode,
                  long cutoff,
                  String keyPrefix)

Description copied from interface: MetadataStore

Same as MetadataStore.prune(PruneMode, long), but with an additional keyPrefix parameter to filter the pruned keys with a prefix.

Specified by:

prune in interface MetadataStore

Parameters:

pruneMode - Prune Mode

cutoff - Oldest time in milliseconds to allow (UTC)

keyPrefix - The prefix for the keys that should be removed

Returns:

the number of pruned entries

toString

public String toString()

Overrides:

toString in class Object

getDiagnostics

public Map<String,String> getDiagnostics()
                                  throws IOException

Description copied from interface: MetadataStore

Get any diagnostics information from a store, as a list of (key, value) tuples for display. Arbitrary values; no guarantee of stability. These are for debugging and testing only.

Specified by:

getDiagnostics in interface MetadataStore

Returns:

a map of strings.

Throws:

IOException - if there is an error

updateParameters

public void updateParameters(Map<String,String> parameters)
                      throws IOException

Description copied from interface: MetadataStore

Tune/update parameters for an existing table.

Specified by:

updateParameters in interface MetadataStore

Parameters:

parameters - map of params to change.

Throws:

IOException - if there is an error

initiateRenameOperation

public RenameTracker initiateRenameOperation(StoreContext storeContext,
                                             org.apache.hadoop.fs.Path source,
                                             S3AFileStatus sourceStatus,
                                             org.apache.hadoop.fs.Path dest)
                                      throws IOException

Description copied from interface: MetadataStore

Start a rename operation.

Specified by:

initiateRenameOperation in interface MetadataStore

Parameters:

storeContext - store context.

source - source path

sourceStatus - status of the source file/dir

dest - destination path.

Returns:

the rename tracker

Throws:

IOException - Failure.

setTtlTimeProvider

public void setTtlTimeProvider(ITtlTimeProvider ttlTimeProvider)

Description copied from interface: MetadataStore

The TtlTimeProvider has to be set during the initialization for the metadatastore, but this method can be used for testing, and change the instance during runtime.

Specified by:

setTtlTimeProvider in interface MetadataStore

addAncestors

public void addAncestors(org.apache.hadoop.fs.Path qualifiedPath,
                         @Nullable
                         BulkOperationState operationState)
                  throws IOException

Description copied from interface: MetadataStore

This adds all new ancestors of a path as directories.

Important: to propagate TTL information, any new ancestors added must have their last updated timestamps set through S3Guard.patchLastUpdated(Collection, ITtlTimeProvider).

Specified by:

addAncestors in interface MetadataStore

Parameters:

qualifiedPath - path to update

operationState - (nullable) operational state for a bulk update

Throws:

IOException - failure