org.apache.hadoop.fs.s3a.impl

Class InternalConstants

java.lang.Object

org.apache.hadoop.fs.s3a.impl.InternalConstants

public final class InternalConstants
extends Object

Internal constants private only to the S3A codebase. Please don't refer to these outside of this module & its tests. If you find you need to then either the code is doing something it should not, or these constants need to be uprated to being public and stable entries.

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_BLOCKSIZE Default blocksize as used in blocksize and FS status queries: 33554432.
static int	MAX_ENTRIES_TO_DELETE The maximum number of entries that can be deleted in any bulk delete call to S3: 1000.
static int	RENAME_PARALLEL_LIMIT This is an arbitrary value: 10.
static int	SC_404 404 error code.
static Set<String>	STANDARD_OPENFILE_KEYS The known keys used in a standard openFile call.
static String	THROTTLE_LOG_NAME Name of the log for throttling events.
static String	X_DIRECTORY Directory marker attribute: see HADOOP-16613.

Method Summary

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

RENAME_PARALLEL_LIMIT

public static final int RENAME_PARALLEL_LIMIT

This is an arbitrary value: 10. It declares how many parallel copy operations in a single rename can be queued before the operation pauses and awaits completion. A very large value wouldn't just starve other threads from performing work, there's a risk that the S3 store itself would throttle operations (which all go to the same shard). It is not currently configurable just to avoid people choosing values which work on a microbenchmark (single rename, no other work, ...) but don't scale well to execution in a large process against a common store, all while separate processes are working with the same shard of storage. It should be a factor of MAX_ENTRIES_TO_DELETE so that all copies will have finished before deletion is contemplated. (There's always a block for that, it just makes more sense to perform the bulk delete after another block of copies have completed).

See Also:

Constant Field Values

MAX_ENTRIES_TO_DELETE

public static final int MAX_ENTRIES_TO_DELETE

The maximum number of entries that can be deleted in any bulk delete call to S3: 1000.

See Also:

Constant Field Values

DEFAULT_BLOCKSIZE

public static final int DEFAULT_BLOCKSIZE

Default blocksize as used in blocksize and FS status queries: 33554432.

See Also:

Constant Field Values

STANDARD_OPENFILE_KEYS

@InterfaceStability.Unstable
public static final Set<String> STANDARD_OPENFILE_KEYS

The known keys used in a standard openFile call. if there's a select marker in there then the keyset used becomes that of the select operation.

SC_404

public static final int SC_404

404 error code.

See Also:

Constant Field Values

THROTTLE_LOG_NAME

public static final String THROTTLE_LOG_NAME

Name of the log for throttling events. Value: "org.apache.hadoop.fs.s3a.throttled".

See Also:

Constant Field Values

X_DIRECTORY

public static final String X_DIRECTORY

Directory marker attribute: see HADOOP-16613. Value: "application/x-directory".

See Also:

Constant Field Values