org.apache.hadoop.fs.s3a.s3guard

Class S3GuardTool

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.s3a.s3guard.S3GuardTool

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable, org.apache.hadoop.util.Tool

Direct Known Subclasses:

BucketTool, MarkerTool, S3GuardTool.BucketInfo

@InterfaceAudience.LimitedPrivate(value="management tools")
 @InterfaceStability.Evolving
public abstract class S3GuardTool
extends org.apache.hadoop.conf.Configured
implements org.apache.hadoop.util.Tool, Closeable

CLI to manage S3Guard Metadata Store.

Some management tools invoke this class directly.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	S3GuardTool.BucketInfo Get info about a bucket and its S3Guard integration status.

Field Summary

Fields

Modifier and Type	Field and Description
static String	AGE_OPTIONS_USAGE
static String	DAYS_FLAG
static String	HOURS_FLAG
static String	META_FLAG
static String	MINUTES_FLAG
static String	SECONDS_FLAG
static String	VERBOSE
static String	WRONG_FILESYSTEM Error String when the wrong FS is used for binding: "Wrong filesystem for ".

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	S3GuardTool(org.apache.hadoop.conf.Configuration conf, int min, int max, String... opts) Constructor a S3Guard tool with HDFS configuration.
protected	S3GuardTool(org.apache.hadoop.conf.Configuration conf, String... opts) Constructor a S3Guard tool with HDFS configuration.

Method Summary

Modifier and Type	Method and Description
protected void	addAgeOptions()
protected static org.apache.hadoop.util.ExitUtil.ExitException	badState(String format, Object... args) Build the exception to raise on a bad store/bucket state.
protected S3AFileSystem	bindFilesystem(org.apache.hadoop.fs.FileSystem bindingFS) Sets the filesystem; it must be an S3A FS instance, or a FilterFS around an S3A Filesystem.
void	close() Close the FS.
protected void	dumpFileSystemStatistics(PrintStream stream) Dump the filesystem Storage Statistics if the FS is not null.
protected static void	errorln()
protected static void	errorln(String x)
protected static void	exit(int status, String text)
protected static org.apache.hadoop.util.ExitUtil.ExitException	exitException(int exitCode, String format, Object... args) Build a exception to throw with a formatted message.
protected org.apache.hadoop.fs.shell.CommandFormat	getCommandFormat()
protected S3AFileSystem	getFilesystem()
abstract String	getName() Return sub-command name.
abstract String	getUsage()
protected void	initS3AFileSystem(String path) Create and initialize a new S3A FileSystem instance.
protected static org.apache.hadoop.util.ExitUtil.ExitException	invalidArgs(String format, Object... args) Build the exception to raise on invalid arguments.
static void	main(String[] args) Main entry point.
protected static org.apache.hadoop.util.ExitUtil.ExitException	notFound(FileNotFoundException e) Handle FileNotFoundException by converting to an exit exception with specific error code.
protected List<String>	parseArgs(String[] args) Parse CLI arguments and returns the position arguments.
protected List<String>	parseArgsWithErrorReporting(String[] args) Process the arguments.
protected static void	println(PrintStream out, String format, Object... args) Print a formatted string followed by a newline to the output stream.
protected void	resetBindings() Reset the store and filesystem bindings.
static int	run(org.apache.hadoop.conf.Configuration conf, String... args) Execute the command with the given arguments.
int	run(String[] args)
abstract int	run(String[] args, PrintStream out) Run the tool, capturing the output (if the tool supports that).
protected static org.apache.hadoop.util.ExitUtil.ExitException	s3guardUnsupported() Crate an exception declaring S3Guard is unsupported.
protected static URI	toUri(String s3Path) Convert a path to a URI, catching any URISyntaxException and converting to an invalid args exception.
protected static org.apache.hadoop.util.ExitUtil.ExitException	userAborted(String format, Object... args) Build the exception to raise on user-aborted action.

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Field Detail

WRONG_FILESYSTEM

@VisibleForTesting
public static final String WRONG_FILESYSTEM

Error String when the wrong FS is used for binding: "Wrong filesystem for ".

See Also:

Constant Field Values

META_FLAG

public static final String META_FLAG

See Also:

Constant Field Values

DAYS_FLAG

public static final String DAYS_FLAG

See Also:

Constant Field Values

HOURS_FLAG

public static final String HOURS_FLAG

See Also:

Constant Field Values

MINUTES_FLAG

public static final String MINUTES_FLAG

See Also:

Constant Field Values

SECONDS_FLAG

public static final String SECONDS_FLAG

See Also:

Constant Field Values

AGE_OPTIONS_USAGE

public static final String AGE_OPTIONS_USAGE

See Also:

Constant Field Values

VERBOSE

public static final String VERBOSE

See Also:

Constant Field Values

Constructor Detail

S3GuardTool

protected S3GuardTool(org.apache.hadoop.conf.Configuration conf,
                      String... opts)

Constructor a S3Guard tool with HDFS configuration.

Parameters:

conf - Configuration.

opts - any boolean options to support

S3GuardTool

protected S3GuardTool(org.apache.hadoop.conf.Configuration conf,
                      int min,
                      int max,
                      String... opts)

Constructor a S3Guard tool with HDFS configuration.

Parameters:

conf - Configuration.

min - min number of args

max - max number of args

opts - any boolean options to support

Method Detail

getUsage

public abstract String getUsage()

getName

public abstract String getName()

Return sub-command name.

Returns:

sub-command name.

close

public void close()
           throws IOException

Close the FS.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException - on failure.

addAgeOptions

protected final void addAgeOptions()

initS3AFileSystem

protected void initS3AFileSystem(String path)
                          throws IOException

Create and initialize a new S3A FileSystem instance.

Parameters:

path - s3a URI

Throws:

IOException - failure to init filesystem

ExitUtil.ExitException - if the FS is not an S3A FS

parseArgs

protected List<String> parseArgs(String[] args)

Parse CLI arguments and returns the position arguments. The options are stored in commandFormat.

Parameters:

args - command line arguments.

Returns:

the position arguments from CLI.

parseArgsWithErrorReporting

protected List<String> parseArgsWithErrorReporting(String[] args)
                                            throws org.apache.hadoop.util.ExitUtil.ExitException

Process the arguments.

Parameters:

args - raw args

Returns:

process arg list.

Throws:

ExitUtil.ExitException - if there's an unknown option.

getFilesystem

protected S3AFileSystem getFilesystem()

bindFilesystem

protected S3AFileSystem bindFilesystem(org.apache.hadoop.fs.FileSystem bindingFS)

Sets the filesystem; it must be an S3A FS instance, or a FilterFS around an S3A Filesystem.

Parameters:

bindingFS - filesystem to bind to

Returns:

the bound FS.

Throws:

ExitUtil.ExitException - if the FS is not an S3 FS

resetBindings

protected void resetBindings()

Reset the store and filesystem bindings.

getCommandFormat

protected final org.apache.hadoop.fs.shell.CommandFormat getCommandFormat()

run

public final int run(String[] args)
              throws Exception

Specified by:

run in interface org.apache.hadoop.util.Tool

Throws:

Exception

run

public abstract int run(String[] args,
                        PrintStream out)
                 throws Exception,
                        org.apache.hadoop.util.ExitUtil.ExitException

Run the tool, capturing the output (if the tool supports that). As well as returning an exit code, the implementations can choose to throw an instance of ExitUtil.ExitException with their exit code set to the desired exit value. The exit code of such an exception is used for the tool's exit code, and the stack trace only logged at debug.

Parameters:

args - argument list

out - output stream

Returns:

the exit code to return.

Throws:

Exception - on any failure

org.apache.hadoop.util.ExitUtil.ExitException

dumpFileSystemStatistics

protected void dumpFileSystemStatistics(PrintStream stream)

Dump the filesystem Storage Statistics if the FS is not null. Only non-zero statistics are printed.

Parameters:

stream - output stream

toUri

protected static URI toUri(String s3Path)

Convert a path to a URI, catching any URISyntaxException and converting to an invalid args exception.

Parameters:

s3Path - path to convert to a URI

Returns:

a URI of the path

Throws:

ExitUtil.ExitException - INVALID_ARGUMENT if the URI is invalid

errorln

protected static void errorln()

errorln

protected static void errorln(String x)

println

protected static void println(PrintStream out,
                              String format,
                              Object... args)

Print a formatted string followed by a newline to the output stream.

Parameters:

out - destination

format - format string

args - optional arguments

notFound

protected static org.apache.hadoop.util.ExitUtil.ExitException notFound(FileNotFoundException e)

Handle FileNotFoundException by converting to an exit exception with specific error code.

Parameters:

e - exception

Returns:

a new exception to throw

invalidArgs

protected static org.apache.hadoop.util.ExitUtil.ExitException invalidArgs(String format,
                                                                           Object... args)

Build the exception to raise on invalid arguments.

Parameters:

format - string format

args - optional arguments for the string

Returns:

a new exception to throw

badState

protected static org.apache.hadoop.util.ExitUtil.ExitException badState(String format,
                                                                        Object... args)

Build the exception to raise on a bad store/bucket state.

Parameters:

format - string format

args - optional arguments for the string

Returns:

a new exception to throw

s3guardUnsupported

protected static org.apache.hadoop.util.ExitUtil.ExitException s3guardUnsupported()

Crate an exception declaring S3Guard is unsupported.

Returns:

an exception raise.

userAborted

protected static org.apache.hadoop.util.ExitUtil.ExitException userAborted(String format,
                                                                           Object... args)

Build the exception to raise on user-aborted action.

Parameters:

format - string format

args - optional arguments for the string

Returns:

a new exception to throw

exitException

protected static org.apache.hadoop.util.ExitUtil.ExitException exitException(int exitCode,
                                                                             String format,
                                                                             Object... args)

Build a exception to throw with a formatted message.

Parameters:

exitCode - exit code to use

format - string format

args - optional arguments for the string

Returns:

a new exception to throw

run

public static int run(org.apache.hadoop.conf.Configuration conf,
                      String... args)
               throws Exception

Execute the command with the given arguments.

Parameters:

conf - Hadoop configuration.

args - command specific arguments.

Returns:

exit code.

Throws:

Exception - on I/O errors.

main

public static void main(String[] args)

Main entry point. Calls System.exit() on all execution paths.

Parameters:

args - argument list

exit

protected static void exit(int status,
                           String text)