| Package | Description |
|---|---|
| org.apache.hadoop.fs.s3a |
S3A Filesystem.
|
| org.apache.hadoop.fs.s3a.s3guard |
This package contains classes related to S3Guard: a feature of S3A to mask
the eventual consistency behavior of S3 and optimize access patterns by
coordinating with a strongly consistent external store for file system
metadata.
|
| Modifier and Type | Method and Description |
|---|---|
MetadataStore |
S3AFileSystem.getMetadataStore()
Get the metadata store.
|
| Modifier and Type | Class and Description |
|---|---|
class |
DynamoDBMetadataStore
DynamoDBMetadataStore is a
MetadataStore that persists
file system metadata to DynamoDB. |
class |
LocalMetadataStore
This is a local, in-memory implementation of MetadataStore.
|
class |
NullMetadataStore
A no-op implementation of MetadataStore.
|
| Modifier and Type | Method and Description |
|---|---|
static MetadataStore |
S3Guard.getMetadataStore(org.apache.hadoop.fs.FileSystem fs)
Create a new instance of the configured MetadataStore.
|
MetadataStore |
S3GuardTool.getStore() |
| Modifier and Type | Method and Description |
|---|---|
static void |
S3Guard.addAncestors(MetadataStore metadataStore,
org.apache.hadoop.fs.Path qualifiedPath,
String username) |
static void |
S3Guard.addMoveAncestors(MetadataStore ms,
Collection<org.apache.hadoop.fs.Path> srcPaths,
Collection<PathMetadata> dstMetas,
org.apache.hadoop.fs.Path srcRoot,
org.apache.hadoop.fs.Path srcPath,
org.apache.hadoop.fs.Path dstPath,
String owner)
Helper method that records the move of all ancestors of a path.
|
static void |
S3Guard.addMoveDir(MetadataStore ms,
Collection<org.apache.hadoop.fs.Path> srcPaths,
Collection<PathMetadata> dstMetas,
org.apache.hadoop.fs.Path srcPath,
org.apache.hadoop.fs.Path dstPath,
String owner)
Helper function that records the move of directory paths, adding
resulting metadata to the supplied lists.
|
static void |
S3Guard.addMoveFile(MetadataStore ms,
Collection<org.apache.hadoop.fs.Path> srcPaths,
Collection<PathMetadata> dstMetas,
org.apache.hadoop.fs.Path srcPath,
org.apache.hadoop.fs.Path dstPath,
long size,
long blockSize,
String owner)
Like
S3Guard.addMoveDir(MetadataStore, Collection, Collection, Path,
Path, String) (), but for files. |
static org.apache.hadoop.fs.FileStatus[] |
S3Guard.dirListingUnion(MetadataStore ms,
org.apache.hadoop.fs.Path path,
List<org.apache.hadoop.fs.FileStatus> backingStatuses,
DirListingMetadata dirMeta,
boolean isAuthoritative)
Given directory listing metadata from both the backing store and the
MetadataStore, merge the two sources of truth to create a consistent
view of the current directory contents, which can be returned to clients.
|
static boolean |
S3Guard.isNullMetadataStore(MetadataStore ms)
Although NullMetadataStore does nothing, callers may wish to avoid work
(fast path) when the NullMetadataStore is in use.
|
static void |
S3Guard.makeDirsOrdered(MetadataStore ms,
List<org.apache.hadoop.fs.Path> dirs,
String owner,
boolean authoritative)
Deprecated.
this is no longer called by
S3AFilesystem.innerMkDirs.
See: HADOOP-15079 (January 2018).
It is currently retained because of its discussion in the method on
atomicity and in case we need to reinstate it or adapt the current
process of directory marker creation.
But it is not being tested and so may age with time...consider
deleting it in future if it's clear there's no need for it. |
protected static void |
S3GuardTool.printStoreDiagnostics(PrintStream out,
MetadataStore store)
Retrieve and Print store diagnostics.
|
static S3AFileStatus |
S3Guard.putAndReturn(MetadataStore ms,
S3AFileStatus status,
S3AInstrumentation instrumentation)
Helper function which puts a given S3AFileStatus into the MetadataStore and
returns the same S3AFileStatus.
|
protected void |
S3GuardTool.setStore(MetadataStore store) |
| Constructor and Description |
|---|
DescendantsIterator(MetadataStore ms,
PathMetadata meta)
Creates a new
DescendantsIterator. |
MetadataStoreListFilesIterator(MetadataStore ms,
PathMetadata meta,
boolean allowAuthoritative) |
Copyright © 2008–2019 Apache Software Foundation. All rights reserved.