org.apache.hadoop.fs.s3a.tools

Class MarkerTool

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.s3a.s3guard.S3GuardTool

org.apache.hadoop.fs.s3a.tools.MarkerTool

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.conf.Configurable, org.apache.hadoop.util.Tool

@InterfaceAudience.LimitedPrivate(value="management tools")
 @InterfaceStability.Unstable
public final class MarkerTool
extends S3GuardTool

Audit and S3 bucket for directory markers.

This tool does not go anywhere near S3Guard; its scan bypasses any metastore as we are explicitly looking for marker objects.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	MarkerTool.MarkerPurgeSummary Result of a call of purgeMarkers(DirMarkerTracker, int); included in MarkerTool.ScanResult so must share visibility.
static class	MarkerTool.ScanArgs Arguments for the scan.
static class	MarkerTool.ScanArgsBuilder Builder of the scan arguments.
static class	MarkerTool.ScanResult Result of the scan operation.

Nested classes/interfaces inherited from class org.apache.hadoop.fs.s3a.s3guard.S3GuardTool

S3GuardTool.BucketInfo

Field Summary

Fields

Modifier and Type	Field and Description
static String	AUDIT Audit sub-command: "-audit".
static String	CLEAN Clean Sub-command: "-clean".
static String	MARKERS Name of this tool: "markers".
static String	OPT_AUDIT Audit sub-command: "audit".
static String	OPT_CLEAN Clean Sub-command: "clean".
static String	OPT_LIMIT Limit of objects to scan: "limit".
static String	OPT_MAX Max number of markers to find: "max".
static String	OPT_MIN Min number of markers to find: "min".
static String	OPT_NONAUTH Only consider markers found in non-authoritative paths as failures: "nonauth".
static String	OPT_OUT Name of a file to save the list of markers to: "out".
static String	PURPOSE Purpose of this tool: "View and manipulate S3 directory markers".
static int	UNLIMITED_LISTING Constant to use when there is no limit on the number of objects listed: 0.
static int	UNLIMITED_MIN_MARKERS Constant to use when there is no minimum number of markers: -1.

Fields inherited from class org.apache.hadoop.fs.s3a.s3guard.S3GuardTool

AGE_OPTIONS_USAGE, CMK_FLAG, DAYS_FLAG, HOURS_FLAG, META_FLAG, MINUTES_FLAG, READ_FLAG, REGION_FLAG, SECONDS_FLAG, SSE_FLAG, TAG_FLAG, VERBOSE, WRITE_FLAG, WRONG_FILESYSTEM

Constructor Summary

Constructors

Constructor and Description
MarkerTool(org.apache.hadoop.conf.Configuration conf) Constructor.

Method Summary

Modifier and Type	Method and Description
static MarkerTool.ScanResult	execMarkerTool(MarkerTool.ScanArgs scanArgs) Execute the marker tool, with no checks on return codes.
String	getName() Return sub-command name.
String	getUsage()
boolean	isVerbose()
void	resetBindings() Reset the store and filesystem bindings.
int	run(String[] args, PrintStream stream) Run the tool, capturing the output (if the tool supports that).
void	setVerbose(boolean verbose)

Methods inherited from class org.apache.hadoop.fs.s3a.s3guard.S3GuardTool

addAgeOptions, badState, bindFilesystem, checkBucketNameOrDDBTableNameProvided, checkIfS3BucketIsGuarded, close, dumpFileSystemStatistics, errorln, errorln, exit, exitException, getCommandFormat, getFilesystem, getStore, initMetadataStore, initS3AFileSystem, invalidArgs, main, parseArgs, parseDynamoDBRegion, println, printStoreDiagnostics, run, run, setStore, storeNotFound, toUri, userAborted

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.conf.Configurable

getConf, setConf

Field Detail

MARKERS

public static final String MARKERS

Name of this tool: "markers".

See Also:

Constant Field Values

PURPOSE

public static final String PURPOSE

Purpose of this tool: "View and manipulate S3 directory markers".

See Also:

Constant Field Values

OPT_AUDIT

public static final String OPT_AUDIT

Audit sub-command: "audit".

See Also:

Constant Field Values

OPT_CLEAN

public static final String OPT_CLEAN

Clean Sub-command: "clean".

See Also:

Constant Field Values

AUDIT

public static final String AUDIT

Audit sub-command: "-audit".

See Also:

Constant Field Values

CLEAN

public static final String CLEAN

Clean Sub-command: "-clean".

See Also:

Constant Field Values

OPT_MIN

public static final String OPT_MIN

Min number of markers to find: "min".

See Also:

Constant Field Values

OPT_MAX

public static final String OPT_MAX

Max number of markers to find: "max".

See Also:

Constant Field Values

OPT_OUT

public static final String OPT_OUT

Name of a file to save the list of markers to: "out".

See Also:

Constant Field Values

OPT_LIMIT

public static final String OPT_LIMIT

Limit of objects to scan: "limit".

See Also:

Constant Field Values

OPT_NONAUTH

public static final String OPT_NONAUTH

Only consider markers found in non-authoritative paths as failures: "nonauth".

See Also:

Constant Field Values

UNLIMITED_LISTING

public static final int UNLIMITED_LISTING

Constant to use when there is no limit on the number of objects listed: 0.

The value is 0 and not -1 because it allows for the limit to be set on the command line -limit 0. The command line parser rejects -limit -1 as the -1 is interpreted as the (unknown) option "-1".

See Also:

Constant Field Values

UNLIMITED_MIN_MARKERS

public static final int UNLIMITED_MIN_MARKERS

Constant to use when there is no minimum number of markers: -1.

See Also:

Constant Field Values

Constructor Detail

MarkerTool

public MarkerTool(org.apache.hadoop.conf.Configuration conf)

Constructor.

Parameters:

conf - configuration

Method Detail

getUsage

public String getUsage()

Specified by:

getUsage in class S3GuardTool

getName

public String getName()

Description copied from class: S3GuardTool

Return sub-command name.

Specified by:

getName in class S3GuardTool

Returns:

sub-command name.

resetBindings

public void resetBindings()

Description copied from class: S3GuardTool

Reset the store and filesystem bindings.

Overrides:

resetBindings in class S3GuardTool

run

public int run(String[] args,
               PrintStream stream)
        throws org.apache.hadoop.util.ExitUtil.ExitException,
               Exception

Description copied from class: S3GuardTool

Run the tool, capturing the output (if the tool supports that). As well as returning an exit code, the implementations can choose to throw an instance of ExitUtil.ExitException with their exit code set to the desired exit value. The exit code of such an exception is used for the tool's exit code, and the stack trace only logged at debug.

Specified by:

run in class S3GuardTool

Parameters:

args - argument list

stream - output stream

Returns:

the exit code to return.

Throws:

Exception - on any failure

org.apache.hadoop.util.ExitUtil.ExitException

isVerbose

public boolean isVerbose()

setVerbose

public void setVerbose(boolean verbose)

execMarkerTool

public static MarkerTool.ScanResult execMarkerTool(MarkerTool.ScanArgs scanArgs)
                                            throws IOException

Execute the marker tool, with no checks on return codes.

Parameters:

scanArgs - set of args for the scanner.

Returns:

the result

Throws:

IOException