org.apache.hadoop.fs.s3a.s3guard

Class S3GuardTool

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.s3a.s3guard.S3GuardTool

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable, org.apache.hadoop.util.Tool

Direct Known Subclasses:

MarkerTool, S3GuardTool.BucketInfo, SelectTool

@InterfaceAudience.LimitedPrivate(value="management tools")
 @InterfaceStability.Evolving
public abstract class S3GuardTool
extends org.apache.hadoop.conf.Configured
implements org.apache.hadoop.util.Tool, Closeable

CLI to manage S3Guard Metadata Store.

Some management tools invoke this class directly.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	S3GuardTool.BucketInfo Get info about a bucket and its S3Guard integration status.

Field Summary

Fields

Modifier and Type	Field and Description
static String	AGE_OPTIONS_USAGE
static String	CMK_FLAG
static String	DAYS_FLAG
static String	HOURS_FLAG
static String	META_FLAG
static String	MINUTES_FLAG
static String	READ_FLAG
static String	REGION_FLAG
static String	SECONDS_FLAG
static String	SSE_FLAG
static String	TAG_FLAG
static String	VERBOSE
static String	WRITE_FLAG
static String	WRONG_FILESYSTEM Error String when the wrong FS is used for binding: "Wrong filesystem for ".

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	S3GuardTool(org.apache.hadoop.conf.Configuration conf, String... opts) Constructor a S3Guard tool with HDFS configuration.

Method Summary

Modifier and Type	Method and Description
protected void	addAgeOptions()
protected static org.apache.hadoop.util.ExitUtil.ExitException	badState(String format, Object... args) Build the exception to raise on a bad store/bucket state.
protected S3AFileSystem	bindFilesystem(org.apache.hadoop.fs.FileSystem bindingFS) Sets the filesystem; it must be an S3A FS instance, or a FilterFS around an S3A Filesystem.
protected void	checkBucketNameOrDDBTableNameProvided(List<String> paths) Check if bucket or DDB table name is set.
protected void	checkIfS3BucketIsGuarded(List<String> paths)
void	close() Close the FS and metastore.
protected void	dumpFileSystemStatistics(PrintStream stream) Dump the filesystem Storage Statistics if the FS is not null.
protected static void	errorln()
protected static void	errorln(String x)
protected static void	exit(int status, String text)
protected static org.apache.hadoop.util.ExitUtil.ExitException	exitException(int exitCode, String format, Object... args) Build a exception to throw with a formatted message.
protected org.apache.hadoop.fs.shell.CommandFormat	getCommandFormat()
protected S3AFileSystem	getFilesystem()
abstract String	getName() Return sub-command name.
MetadataStore	getStore()
abstract String	getUsage()
protected MetadataStore	initMetadataStore(boolean forceCreate) Parse metadata store from command line option or HDFS configuration.
protected void	initS3AFileSystem(String path) Create and initialize a new S3A FileSystem instance.
protected static org.apache.hadoop.util.ExitUtil.ExitException	invalidArgs(String format, Object... args) Build the exception to raise on invalid arguments.
static void	main(String[] args) Main entry point.
protected List<String>	parseArgs(String[] args) Parse CLI arguments and returns the position arguments.
protected void	parseDynamoDBRegion(List<String> paths) Parse DynamoDB region from either -m option or a S3 path.
protected static void	println(PrintStream out, String format, Object... args) Print a formatted string followed by a newline to the output stream.
protected static void	printStoreDiagnostics(PrintStream out, MetadataStore store) Retrieve and Print store diagnostics.
protected void	resetBindings() Reset the store and filesystem bindings.
static int	run(org.apache.hadoop.conf.Configuration conf, String... args) Execute the command with the given arguments.
int	run(String[] args)
abstract int	run(String[] args, PrintStream out) Run the tool, capturing the output (if the tool supports that).
protected void	setStore(MetadataStore store)
protected static org.apache.hadoop.util.ExitUtil.ExitException	storeNotFound(FileNotFoundException e) Handle store not found by converting to an exit exception with specific error code.
protected static URI	toUri(String s3Path) Convert a path to a URI, catching any URISyntaxException and converting to an invalid args exception.
protected static org.apache.hadoop.util.ExitUtil.ExitException	userAborted(String format, Object... args) Build the exception to raise on user-aborted action.

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Field Detail

WRONG_FILESYSTEM

public static final String WRONG_FILESYSTEM

Error String when the wrong FS is used for binding: "Wrong filesystem for ".

See Also:

Constant Field Values

META_FLAG

public static final String META_FLAG

See Also:

Constant Field Values

DAYS_FLAG

public static final String DAYS_FLAG

See Also:

Constant Field Values

HOURS_FLAG

public static final String HOURS_FLAG

See Also:

Constant Field Values

MINUTES_FLAG

public static final String MINUTES_FLAG

See Also:

Constant Field Values

SECONDS_FLAG

public static final String SECONDS_FLAG

See Also:

Constant Field Values

AGE_OPTIONS_USAGE

public static final String AGE_OPTIONS_USAGE

See Also:

Constant Field Values

REGION_FLAG

public static final String REGION_FLAG

See Also:

Constant Field Values

READ_FLAG

public static final String READ_FLAG

See Also:

Constant Field Values

WRITE_FLAG

public static final String WRITE_FLAG

See Also:

Constant Field Values

SSE_FLAG

public static final String SSE_FLAG

See Also:

Constant Field Values

CMK_FLAG

public static final String CMK_FLAG

See Also:

Constant Field Values

TAG_FLAG

public static final String TAG_FLAG

See Also:

Constant Field Values

VERBOSE

public static final String VERBOSE

See Also:

Constant Field Values

Constructor Detail

S3GuardTool

protected S3GuardTool(org.apache.hadoop.conf.Configuration conf,
                      String... opts)

Constructor a S3Guard tool with HDFS configuration.

Parameters:

conf - Configuration.

opts - any boolean options to support

Method Detail

getUsage

public abstract String getUsage()

getName

public abstract String getName()

Return sub-command name.

Returns:

sub-command name.

close

public void close()
           throws IOException

Close the FS and metastore.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException - on failure.

parseDynamoDBRegion

protected void parseDynamoDBRegion(List<String> paths)
                            throws IOException

Parse DynamoDB region from either -m option or a S3 path. Note that as a side effect, if the paths included an S3 path, and there is no region set on the CLI, then the S3A FS is initialized, after which filesystem will no longer be null.

Parameters:

paths - remaining parameters from CLI.

Throws:

IOException - on I/O errors.

ExitUtil.ExitException - on validation errors

addAgeOptions

protected void addAgeOptions()

checkIfS3BucketIsGuarded

protected void checkIfS3BucketIsGuarded(List<String> paths)
                                 throws IOException

Throws:

IOException

checkBucketNameOrDDBTableNameProvided

protected void checkBucketNameOrDDBTableNameProvided(List<String> paths)

Check if bucket or DDB table name is set.

Parameters:

paths - position arguments in which S3 path is provided.

initMetadataStore

protected MetadataStore initMetadataStore(boolean forceCreate)
                                   throws IOException

Parse metadata store from command line option or HDFS configuration.

Parameters:

forceCreate - override the auto-creation setting to true.

Returns:

a initialized metadata store.

Throws:

IOException - on unsupported metadata store.

initS3AFileSystem

protected void initS3AFileSystem(String path)
                          throws IOException

Create and initialize a new S3A FileSystem instance. This instance is always created without S3Guard, so allowing a previously created metastore to be patched in. Note: this is a bit convoluted as it needs to also handle the situation of a per-bucket option in core-site.xml, which isn't easily overridden. The new config and the setting of the values before any Configuration.get() calls are critical.

Parameters:

path - s3a URI

Throws:

IOException - failure to init filesystem

ExitUtil.ExitException - if the FS is not an S3A FS

parseArgs

protected List<String> parseArgs(String[] args)

Parse CLI arguments and returns the position arguments. The options are stored in commandFormat.

Parameters:

args - command line arguments.

Returns:

the position arguments from CLI.

getFilesystem

protected S3AFileSystem getFilesystem()

bindFilesystem

protected S3AFileSystem bindFilesystem(org.apache.hadoop.fs.FileSystem bindingFS)

Sets the filesystem; it must be an S3A FS instance, or a FilterFS around an S3A Filesystem.

Parameters:

bindingFS - filesystem to bind to

Returns:

the bound FS.

Throws:

ExitUtil.ExitException - if the FS is not an S3 FS

getStore

public MetadataStore getStore()

setStore

protected void setStore(MetadataStore store)

resetBindings

protected void resetBindings()

Reset the store and filesystem bindings.

getCommandFormat

protected org.apache.hadoop.fs.shell.CommandFormat getCommandFormat()

run

public final int run(String[] args)
              throws Exception

Specified by:

run in interface org.apache.hadoop.util.Tool

Throws:

Exception

run

public abstract int run(String[] args,
                        PrintStream out)
                 throws Exception,
                        org.apache.hadoop.util.ExitUtil.ExitException

Run the tool, capturing the output (if the tool supports that). As well as returning an exit code, the implementations can choose to throw an instance of ExitUtil.ExitException with their exit code set to the desired exit value. The exit code of such an exception is used for the tool's exit code, and the stack trace only logged at debug.

Parameters:

args - argument list

out - output stream

Returns:

the exit code to return.

Throws:

Exception - on any failure

org.apache.hadoop.util.ExitUtil.ExitException

dumpFileSystemStatistics

protected void dumpFileSystemStatistics(PrintStream stream)

Dump the filesystem Storage Statistics if the FS is not null. Only non-zero statistics are printed.

Parameters:

stream - output stream

toUri

protected static URI toUri(String s3Path)

Convert a path to a URI, catching any URISyntaxException and converting to an invalid args exception.

Parameters:

s3Path - path to convert to a URI

Returns:

a URI of the path

Throws:

ExitUtil.ExitException - INVALID_ARGUMENT if the URI is invalid

errorln

protected static void errorln()

errorln

protected static void errorln(String x)

println

protected static void println(PrintStream out,
                              String format,
                              Object... args)

Print a formatted string followed by a newline to the output stream.

Parameters:

out - destination

format - format string

args - optional arguments

printStoreDiagnostics

protected static void printStoreDiagnostics(PrintStream out,
                                            MetadataStore store)
                                     throws IOException

Retrieve and Print store diagnostics.

Parameters:

out - output stream

store - store

Throws:

IOException - Failure to retrieve the data.

storeNotFound

protected static org.apache.hadoop.util.ExitUtil.ExitException storeNotFound(FileNotFoundException e)

Handle store not found by converting to an exit exception with specific error code.

Parameters:

e - exception

Returns:

a new exception to throw

invalidArgs

protected static org.apache.hadoop.util.ExitUtil.ExitException invalidArgs(String format,
                                                                           Object... args)

Build the exception to raise on invalid arguments.

Parameters:

format - string format

args - optional arguments for the string

Returns:

a new exception to throw

badState

protected static org.apache.hadoop.util.ExitUtil.ExitException badState(String format,
                                                                        Object... args)

Build the exception to raise on a bad store/bucket state.

Parameters:

format - string format

args - optional arguments for the string

Returns:

a new exception to throw

userAborted

protected static org.apache.hadoop.util.ExitUtil.ExitException userAborted(String format,
                                                                           Object... args)

Build the exception to raise on user-aborted action.

Parameters:

format - string format

args - optional arguments for the string

Returns:

a new exception to throw

exitException

protected static org.apache.hadoop.util.ExitUtil.ExitException exitException(int exitCode,
                                                                             String format,
                                                                             Object... args)

Build a exception to throw with a formatted message.

Parameters:

exitCode - exit code to use

format - string format

args - optional arguments for the string

Returns:

a new exception to throw

run

public static int run(org.apache.hadoop.conf.Configuration conf,
                      String... args)
               throws Exception

Execute the command with the given arguments.

Parameters:

conf - Hadoop configuration.

args - command specific arguments.

Returns:

exit code.

Throws:

Exception - on I/O errors.

main

public static void main(String[] args)

Main entry point. Calls System.exit() on all execution paths.

Parameters:

args - argument list

exit

protected static void exit(int status,
                           String text)