org.apache.hadoop.fs.s3a.s3guard

Class S3GuardFsck

java.lang.Object

org.apache.hadoop.fs.s3a.s3guard.S3GuardFsck

public class S3GuardFsck
extends Object

Main class for the FSCK factored out from S3GuardTool The implementation uses fixed DynamoDBMetadataStore as the backing store for metadata. Functions:

• Checking metadata consistency between S3 and metadatastore
• Checking the internal metadata consistency

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	S3GuardFsck.ComparePair A compare pair with the pair of metadata and the list of violations.
static class	S3GuardFsck.DDBTree DDBTree is the tree that represents the structure of items in the DynamoDB.
static class	S3GuardFsck.Violation Violation with severity and the handler.

Field Summary

Fields

Modifier and Type	Field and Description
static String	ROOT_PATH_STRING

Constructor Summary

Constructors

Constructor and Description
S3GuardFsck(S3AFileSystem fs, MetadataStore ms) Creates an S3GuardFsck.

Method Summary

Modifier and Type	Method and Description
List<S3GuardFsck.ComparePair>	checkDdbInternalConsistency(org.apache.hadoop.fs.Path basePath) Check the DynamoDB metadatastore internally for consistency.
protected S3GuardFsck.ComparePair	compareFileStatusToPathMetadata(S3AFileStatus s3FileStatus, PathMetadata msPathMetadata) Compares a S3AFileStatus from S3 to a PathMetadata from the metadata store.
protected List<S3GuardFsck.ComparePair>	compareS3DirContentToMs(S3AFileStatus s3CurrentDir, List<S3AFileStatus> children) Compares S3 directory content to the metadata store.
List<S3GuardFsck.ComparePair>	compareS3ToMs(org.apache.hadoop.fs.Path p) Compares S3 to MS.
void	fixViolations(List<S3GuardFsck.ComparePair> violations) Fix violations found during check.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

ROOT_PATH_STRING

public static final String ROOT_PATH_STRING

See Also:

Constant Field Values

Constructor Detail

S3GuardFsck

public S3GuardFsck(S3AFileSystem fs,
                   MetadataStore ms)
            throws InvalidParameterException

Creates an S3GuardFsck.

Parameters:

fs - the filesystem to compare to

ms - metadatastore the metadatastore to compare with (dynamo)

Throws:

InvalidParameterException

Method Detail

compareS3ToMs

public List<S3GuardFsck.ComparePair> compareS3ToMs(org.apache.hadoop.fs.Path p)
                                            throws IOException

Compares S3 to MS. Iterative breadth first walk on the S3 structure from a given root. Creates a list of pairs (metadata in S3 and in the MetadataStore) where the consistency or any rule is violated. Uses S3GuardFsckViolationHandler to handle violations. The violations are listed in Enums: S3GuardFsck.Violation

Parameters:

p - the root path to start the traversal

Returns:

a list of S3GuardFsck.ComparePair

Throws:

IOException

compareS3DirContentToMs

protected List<S3GuardFsck.ComparePair> compareS3DirContentToMs(S3AFileStatus s3CurrentDir,
                                                                List<S3AFileStatus> children)
                                                         throws IOException

Compares S3 directory content to the metadata store.

Parameters:

s3CurrentDir - file status of the current directory

children - the contents of the directory

Returns:

the compare pairs with violations of consistency

Throws:

IOException

compareFileStatusToPathMetadata

protected S3GuardFsck.ComparePair compareFileStatusToPathMetadata(S3AFileStatus s3FileStatus,
                                                                  PathMetadata msPathMetadata)
                                                           throws IOException

Compares a S3AFileStatus from S3 to a PathMetadata from the metadata store. Finds violated invariants and consistency issues.

Parameters:

s3FileStatus - the file status from S3

msPathMetadata - the path metadata from metadatastore

Returns:

S3GuardFsck.ComparePair with the found issues

Throws:

IOException

fixViolations

public void fixViolations(List<S3GuardFsck.ComparePair> violations)
                   throws IOException

Fix violations found during check. Currently only supports handling the following violation: - Violation.ORPHAN_DDB_ENTRY

Parameters:

violations - to be handled

Throws:

IOException - throws the error if there's any during handling

checkDdbInternalConsistency

public List<S3GuardFsck.ComparePair> checkDdbInternalConsistency(org.apache.hadoop.fs.Path basePath)
                                                          throws IOException

Check the DynamoDB metadatastore internally for consistency.

 Tasks to do here:
  - find orphan entries (entries without a parent).
  - find if a file's parent is not a directory (so the parent is a file).
  - find entries where the parent is a tombstone.
  - warn: no lastUpdated field.
 

Throws:

IOException