org.apache.hadoop.fs.s3a.s3guard

Class DumpS3GuardDynamoTable

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.hadoop.service.launcher.AbstractLaunchableService

org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardDynamoDBDiagnostic

org.apache.hadoop.fs.s3a.s3guard.DumpS3GuardDynamoTable

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.service.launcher.LaunchableService, org.apache.hadoop.service.Service

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public class DumpS3GuardDynamoTable
extends AbstractS3GuardDynamoDBDiagnostic

This is a low-level diagnostics entry point which does a CVE/TSV dump of the DDB state. As it also lists the filesystem, it actually changes the state of the store during the operation.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

org.apache.hadoop.service.Service.STATE

Field Summary

Fields

Modifier and Type	Field and Description
static String	FLAT_CSV Suffix for the flat list: "-flat.csv".
static String	NAME Application name.
static String	RAW_CSV Suffix for the raw S3 dump: "-s3.csv".
static String	SCAN_CSV Suffix for the DDB scan: "-scan.csv".
static String	SCAN2_CSV Suffix for the second DDB scan: : "-scan-2.csv".
static String	STORE_CSV Suffix for a recursive treewalk through the metastore: "-store.csv".
static String	TREE_CSV Suffix for the treewalk scan of the S3A Filesystem: "-tree.csv".

Constructor Summary

Constructors

Constructor and Description
DumpS3GuardDynamoTable() Instantiate with default name.
DumpS3GuardDynamoTable(S3AFileSystem fs, DynamoDBMetadataStore store, File destFile, URI uri) Bind to a specific FS + store.
DumpS3GuardDynamoTable(String name) Instantiate.

Method Summary

Modifier and Type	Method and Description
protected void	dumpMetastore(org.apache.hadoop.fs.s3a.s3guard.DumpS3GuardDynamoTable.CsvFile csv, org.apache.hadoop.fs.Path basePath) list children under the metastore from a base path, through a recursive query + walk strategy.
protected long	dumpRawS3ObjectStore(org.apache.hadoop.fs.s3a.s3guard.DumpS3GuardDynamoTable.CsvFile csv) Dump the raw S3 Object Store.
static DumpS3GuardDynamoTable	dumpStore(S3AFileSystem fs, DynamoDBMetadataStore store, org.apache.hadoop.conf.Configuration conf, File destFile, URI uri) Entry point to dump the metastore and s3 store world views
int	execute() Dump the filesystem and the metastore.
long	getListStatusCount()
long	getRawObjectStoreCount()
org.apache.commons.lang3.tuple.Pair<Long,Long>	getScanEntryResult()
org.apache.commons.lang3.tuple.Pair<Long,Long>	getSecondScanResult()
long	getTreewalkCount()
protected long	listStatusFilesystem(org.apache.hadoop.fs.s3a.s3guard.DumpS3GuardDynamoTable.CsvFile csv, org.apache.hadoop.fs.Path path) Dump the filesystem via a recursive listStatus call.
static void	main(String[] args) This is the JVM entry point for the service launcher.
protected void	serviceStart() Bind to the argument list, including validating the CLI.
protected long	treewalkFilesystem(org.apache.hadoop.fs.s3a.s3guard.DumpS3GuardDynamoTable.CsvFile csv, org.apache.hadoop.fs.Path base) Dump the filesystem via a treewalk.

Methods inherited from class org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardDynamoDBDiagnostic

bindArgs, bindFromCLI, failure, failure, getArgumentList, getArguments, getFilesystem, getStore, getUri, require, setUri, setUri

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, serviceInit, serviceStop, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.service.Service

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, registerServiceListener, start, stop, unregisterServiceListener, waitForServiceToStop

Field Detail

NAME

public static final String NAME

Application name.

See Also:

Constant Field Values

FLAT_CSV

public static final String FLAT_CSV

Suffix for the flat list: "-flat.csv".

See Also:

Constant Field Values

RAW_CSV

public static final String RAW_CSV

Suffix for the raw S3 dump: "-s3.csv".

See Also:

Constant Field Values

SCAN_CSV

public static final String SCAN_CSV

Suffix for the DDB scan: "-scan.csv".

See Also:

Constant Field Values

SCAN2_CSV

public static final String SCAN2_CSV

Suffix for the second DDB scan: : "-scan-2.csv".

See Also:

Constant Field Values

TREE_CSV

public static final String TREE_CSV

Suffix for the treewalk scan of the S3A Filesystem: "-tree.csv".

See Also:

Constant Field Values

STORE_CSV

public static final String STORE_CSV

Suffix for a recursive treewalk through the metastore: "-store.csv".

See Also:

Constant Field Values

Constructor Detail

DumpS3GuardDynamoTable

public DumpS3GuardDynamoTable(String name)

Instantiate.

Parameters:

name - application name.

DumpS3GuardDynamoTable

public DumpS3GuardDynamoTable()

Instantiate with default name.

DumpS3GuardDynamoTable

public DumpS3GuardDynamoTable(S3AFileSystem fs,
                              DynamoDBMetadataStore store,
                              File destFile,
                              URI uri)

Bind to a specific FS + store.

Parameters:

fs - filesystem

store - metastore to use

destFile - the base filename for output

uri - URI of store -only needed if FS is null.

Method Detail

serviceStart

protected void serviceStart()
                     throws Exception

Bind to the argument list, including validating the CLI.

Overrides:

serviceStart in class org.apache.hadoop.service.AbstractService

Throws:

Exception - failure.

execute

public int execute()
            throws org.apache.hadoop.service.launcher.ServiceLaunchException,
                   IOException

Dump the filesystem and the metastore.

Specified by:

execute in interface org.apache.hadoop.service.launcher.LaunchableService

Overrides:

execute in class org.apache.hadoop.service.launcher.AbstractLaunchableService

Returns:

the exit code.

Throws:

org.apache.hadoop.service.launcher.ServiceLaunchException - on failure.

IOException - IO failure.

treewalkFilesystem

protected long treewalkFilesystem(org.apache.hadoop.fs.s3a.s3guard.DumpS3GuardDynamoTable.CsvFile csv,
                                  org.apache.hadoop.fs.Path base)
                           throws IOException

Dump the filesystem via a treewalk. If metastore entries mark directories as deleted, this walk will not explore them.

Parameters:

csv - destination.

base - base path.

Returns:

number of entries found.

Throws:

IOException - IO failure.

listStatusFilesystem

protected long listStatusFilesystem(org.apache.hadoop.fs.s3a.s3guard.DumpS3GuardDynamoTable.CsvFile csv,
                                    org.apache.hadoop.fs.Path path)
                             throws IOException

Dump the filesystem via a recursive listStatus call.

Parameters:

csv - destination.

Returns:

number of entries found.

Throws:

IOException - IO failure.

dumpRawS3ObjectStore

protected long dumpRawS3ObjectStore(org.apache.hadoop.fs.s3a.s3guard.DumpS3GuardDynamoTable.CsvFile csv)
                             throws IOException

Dump the raw S3 Object Store.

Parameters:

csv - destination.

Returns:

number of entries found.

Throws:

IOException - IO failure.

dumpMetastore

protected void dumpMetastore(org.apache.hadoop.fs.s3a.s3guard.DumpS3GuardDynamoTable.CsvFile csv,
                             org.apache.hadoop.fs.Path basePath)
                      throws IOException

list children under the metastore from a base path, through a recursive query + walk strategy.

Parameters:

csv - dest

basePath - base path

Throws:

IOException - failure.

getScanEntryResult

public org.apache.commons.lang3.tuple.Pair<Long,Long> getScanEntryResult()

getSecondScanResult

public org.apache.commons.lang3.tuple.Pair<Long,Long> getSecondScanResult()

getRawObjectStoreCount

public long getRawObjectStoreCount()

getListStatusCount

public long getListStatusCount()

getTreewalkCount

public long getTreewalkCount()

main

public static void main(String[] args)

This is the JVM entry point for the service launcher. Converts the arguments to a list, then invokes serviceMain(List, AbstractS3GuardDynamoDBDiagnostic).

Parameters:

args - command line arguments.

dumpStore

public static DumpS3GuardDynamoTable dumpStore(@Nullable
                                               S3AFileSystem fs,
                                               @Nullable
                                               DynamoDBMetadataStore store,
                                               @Nullable
                                               org.apache.hadoop.conf.Configuration conf,
                                               File destFile,
                                               @Nullable
                                               URI uri)
                                        throws org.apache.hadoop.util.ExitUtil.ExitException

Entry point to dump the metastore and s3 store world views

Both the FS and the store will be dumped: the store is scanned before and after the sequence to show what changes were made to the store during the list operation.

Parameters:

fs - fs to dump. If null a store must be provided.

store - store to dump (fallback to FS)

conf - configuration to use (fallback to fs)

destFile - base name of the output files.

uri - URI of store -only needed if FS is null.

Returns:

the store

Throws:

ExitUtil.ExitException - failure.