S3AUtils (Apache Hadoop Amazon Web Services support 3.3.4 API)

java.lang.Object
- org.apache.hadoop.fs.s3a.S3AUtils

@InterfaceAudience.Private
 @InterfaceStability.Evolving
public final class S3AUtils
extends Object

Utility methods for S3A code.

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`static interface`	`S3AUtils.CallOnLocatedFileStatus` An interface for use in lambda-expressions working with directory tree listings.
`static interface`	`S3AUtils.LocatedFileStatusMap<T>` An interface for use in lambda-expressions working with directory tree listings.

Field Summary

Fields
Modifier and Type	Field and Description
`static org.apache.hadoop.fs.PathFilter`	`ACCEPT_ALL` A Path filter which accepts all filenames.
`static String`	`E_FORBIDDEN_AWS_PROVIDER` Error message when the AWS provider list built up contains a forbidden entry.
`static String`	`EOF_MESSAGE_IN_XML_PARSER`
`static String`	`EOF_READ_DIFFERENT_LENGTH`
`static org.apache.hadoop.fs.PathFilter`	`HIDDEN_FILE_FILTER` Path filter which ignores any file which starts with .
`static String`	`SSE_C_NO_KEY_ERROR` Encryption SSE-C used but the config lacks an encryption key.
`static String`	`SSE_S3_WITH_KEY_ERROR` Encryption SSE-S3 is used but the caller also set an encryption key.
`static List<Class<?>>`	`STANDARD_AWS_PROVIDERS` The standard AWS provider list for AWS connections.

Method Summary

All Methods Static Methods Concrete Methods Deprecated Methods
Modifier and Type	Method and Description
`static long`	`applyLocatedFiles(org.apache.hadoop.fs.RemoteIterator<? extends org.apache.hadoop.fs.LocatedFileStatus> iterator, S3AUtils.CallOnLocatedFileStatus eval)` Apply an operation to every `LocatedFileStatus` in a remote iterator.
`static AWSCredentialProviderList`	`buildAWSProviderList(URI binding, org.apache.hadoop.conf.Configuration conf, String key, List<Class<?>> defaultValues, Set<Class<?>> forbidden)` Load list of AWS credential provider/credential provider factory classes; support a forbidden list to prevent loops, mandate full secrets, etc.
`static EncryptionSecrets`	`buildEncryptionSecrets(String bucket, org.apache.hadoop.conf.Configuration conf)` Get the server-side encryption or client side encryption algorithm.
`static void`	`clearBucketOption(org.apache.hadoop.conf.Configuration conf, String bucket, String genericKey)` Clear a bucket-specific property.
`static void`	`closeAll(org.slf4j.Logger log, Closeable... closeables)` Deprecated.
`static void`	`closeAutocloseables(org.slf4j.Logger log, AutoCloseable... closeables)` Close the Closeable objects and ignore any Exception or null pointers.
`static com.amazonaws.ClientConfiguration`	`createAwsConf(org.apache.hadoop.conf.Configuration conf, String bucket)` Deprecated. use `createAwsConf(Configuration, String, String)`
`static com.amazonaws.ClientConfiguration`	`createAwsConf(org.apache.hadoop.conf.Configuration conf, String bucket, String awsServiceIdentifier)` Create a new AWS `ClientConfiguration`.
`static AWSCredentialProviderList`	`createAWSCredentialProviderSet(URI binding, org.apache.hadoop.conf.Configuration conf)` Create the AWS credentials from the providers, the URI and the key `Constants.AWS_CREDENTIALS_PROVIDER` in the configuration.
`static S3AFileStatus`	`createFileStatus(org.apache.hadoop.fs.Path keyPath, com.amazonaws.services.s3.model.S3ObjectSummary summary, long blockSize, String owner, String eTag, String versionId, boolean isCSEEnabled)` Create a files status instance from a listing.
`static S3AFileStatus`	`createUploadFileStatus(org.apache.hadoop.fs.Path keyPath, boolean isDir, long size, long blockSize, String owner, String eTag, String versionId)` Create a file status for object we just uploaded.
`static long`	`dateToLong(Date date)` Date to long conversion.
`static void`	`deleteQuietly(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, boolean recursive)` Delete a path quietly: failures are logged at DEBUG.
`static void`	`deleteWithWarning(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, boolean recursive)` Delete a path: failures are logged at WARN.
`static int`	`ensureOutputParameterInRange(String name, long size)` Ensure that the long value is in the range of an integer.
`static IOException`	`extractException(String operation, String path, ExecutionException ee)` Extract an exception from a failed future, and convert to an IOE.
`static <T> List<T>`	`flatmapLocatedFiles(org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> iterator, S3AUtils.LocatedFileStatusMap<Optional<T>> eval)` Map an operation to every `LocatedFileStatus` in a remote iterator, returning a list of the all results which were not empty.
`static S3xLoginHelper.Login`	`getAWSAccessKeys(URI name, org.apache.hadoop.conf.Configuration conf)` Return the access key and secret for S3 API use.
`static String`	`getBucketOption(org.apache.hadoop.conf.Configuration conf, String bucket, String genericKey)` Get a bucket-specific property.
`static S3AEncryptionMethods`	`getEncryptionAlgorithm(String bucket, org.apache.hadoop.conf.Configuration conf)` Get the server-side encryption or client side encryption algorithm.
`static long`	`getMultipartSizeProperty(org.apache.hadoop.conf.Configuration conf, String property, long defVal)` Get a size property from the configuration: this property must be at least equal to `Constants.MULTIPART_MIN_SIZE`.
`static String`	`getS3EncryptionKey(String bucket, org.apache.hadoop.conf.Configuration conf)` Get any S3 encryption key, without propagating exceptions from JCEKs files.
`static String`	`getS3EncryptionKey(String bucket, org.apache.hadoop.conf.Configuration conf, boolean propagateExceptions)` Get any SSE/CSE key from a configuration/credential provider.
`static void`	`initConnectionSettings(org.apache.hadoop.conf.Configuration conf, com.amazonaws.ClientConfiguration awsConf)` Initializes all AWS SDK settings related to connection management.
`static void`	`initProxySupport(org.apache.hadoop.conf.Configuration conf, String bucket, com.amazonaws.ClientConfiguration awsConf)` Initializes AWS SDK proxy support in the AWS client configuration if the S3A settings enable it.
`static int`	`intOption(org.apache.hadoop.conf.Configuration conf, String key, int defVal, int min)` Get a integer option >= the minimum allowed value.
`static boolean`	`isMessageTranslatableToEOF(com.amazonaws.SdkBaseException ex)` Cue that an AWS exception is likely to be an EOF Exception based on the message coming back from the client.
`static boolean`	`isThrottleException(Exception ex)` Is the exception an instance of a throttling exception.
`static S3AFileStatus[]`	`iteratorToStatuses(org.apache.hadoop.fs.RemoteIterator<S3AFileStatus> iterator, Set<org.apache.hadoop.fs.Path> tombstones)` Convert the data of an iterator of `S3AFileStatus` to an array.
`static List<org.apache.hadoop.fs.LocatedFileStatus>`	`listAndFilter(org.apache.hadoop.fs.FileSystem fileSystem, org.apache.hadoop.fs.Path path, boolean recursive, org.apache.hadoop.fs.PathFilter filter)` List located files and filter them as a classic listFiles(path, filter) would do.
`static List<Class<?>>`	`loadAWSProviderClasses(org.apache.hadoop.conf.Configuration conf, String key, Class<?>... defaultValue)` Load list of AWS credential provider/credential provider factory classes.
`static long`	`longBytesOption(org.apache.hadoop.conf.Configuration conf, String key, long defVal, long min)` Get a long option >= the minimum allowed value, supporting memory prefixes K,M,G,T,P.
`static long`	`longOption(org.apache.hadoop.conf.Configuration conf, String key, long defVal, long min)` Get a long option >= the minimum allowed value.
`static String`	`lookupPassword(String bucket, org.apache.hadoop.conf.Configuration conf, String baseKey)` Get a password from a configuration, including JCEKS files, handling both the absolute key and bucket override.
`static String`	`lookupPassword(String bucket, org.apache.hadoop.conf.Configuration conf, String baseKey, String overrideVal)` Deprecated.
`static String`	`lookupPassword(String bucket, org.apache.hadoop.conf.Configuration conf, String baseKey, String overrideVal, String defVal)` Get a password from a configuration, including JCEKS files, handling both the absolute key and bucket override.
`static <T> List<T>`	`mapLocatedFiles(org.apache.hadoop.fs.RemoteIterator<? extends org.apache.hadoop.fs.LocatedFileStatus> iterator, S3AUtils.LocatedFileStatusMap<T> eval)` Map an operation to every `LocatedFileStatus` in a remote iterator, returning a list of the results.
`static <T> Optional<T>`	`maybe(boolean include, T value)` Convert a value into a non-empty Optional instance if the value of `include` is true.
`static String`	`maybeAddTrailingSlash(String key)` Turns a path (relative or otherwise) into an S3 key, adding a trailing "/" if the path is not the root and does not already have a "/" at the end.
`static boolean`	`objectRepresentsDirectory(String name)` Predicate: does the object represent a directory?.
`static org.apache.hadoop.conf.Configuration`	`propagateBucketOptions(org.apache.hadoop.conf.Configuration source, String bucket)` Propagates bucket-specific settings into generic S3A configuration keys.
`static void`	`setBucketOption(org.apache.hadoop.conf.Configuration conf, String bucket, String genericKey, String value)` Set a bucket-specific property to a particular value.
`static boolean`	`setIfDefined(org.apache.hadoop.conf.Configuration config, String key, String val, String origin)` Set a key if the value is non-empty.
`static String`	`stringify(com.amazonaws.services.s3.model.AmazonS3Exception e)` Get low level details of an amazon exception for logging; multi-line.
`static String`	`stringify(com.amazonaws.AmazonServiceException e)` Get low level details of an amazon exception for logging; multi-line.
`static String`	`stringify(com.amazonaws.services.s3.model.S3ObjectSummary summary)` String information about a summary entry for debug messages.
`static IOException`	`translateDynamoDBException(String path, String message, com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException ddbException)` Translate a DynamoDB exception into an IOException.
`static IOException`	`translateException(String operation, org.apache.hadoop.fs.Path path, com.amazonaws.AmazonClientException exception)` Translate an exception raised in an operation into an IOException.
`static IOException`	`translateException(String operation, String path, com.amazonaws.SdkBaseException exception)` Translate an exception raised in an operation into an IOException.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - SSE_C_NO_KEY_ERROR
```
public static final String SSE_C_NO_KEY_ERROR
```
    Encryption SSE-C used but the config lacks an encryption key.
  - SSE_S3_WITH_KEY_ERROR
```
public static final String SSE_S3_WITH_KEY_ERROR
```
    Encryption SSE-S3 is used but the caller also set an encryption key.
  - EOF_MESSAGE_IN_XML_PARSER
```
public static final String EOF_MESSAGE_IN_XML_PARSER
```
    See Also:
    
    Constant Field Values
  - EOF_READ_DIFFERENT_LENGTH
```
public static final String EOF_READ_DIFFERENT_LENGTH
```
    See Also:
    
    Constant Field Values
  - E_FORBIDDEN_AWS_PROVIDER
```
public static final String E_FORBIDDEN_AWS_PROVIDER
```
    Error message when the AWS provider list built up contains a forbidden entry.
    
    See Also:
    
    Constant Field Values
  - STANDARD_AWS_PROVIDERS
```
public static final List<Class<?>> STANDARD_AWS_PROVIDERS
```
    The standard AWS provider list for AWS connections.
  - HIDDEN_FILE_FILTER
```
public static final org.apache.hadoop.fs.PathFilter HIDDEN_FILE_FILTER
```
    Path filter which ignores any file which starts with . or _.
  - ACCEPT_ALL
```
public static final org.apache.hadoop.fs.PathFilter ACCEPT_ALL
```
    A Path filter which accepts all filenames.
- Method Detail
  - translateException
```
public static IOException translateException(String operation,
                                             org.apache.hadoop.fs.Path path,
                                             com.amazonaws.AmazonClientException exception)
```
    Translate an exception raised in an operation into an IOException. The specific type of IOException depends on the class of AmazonClientException passed in, and any status codes included in the operation. That is: HTTP error codes are examined and can be used to build a more specific response.
    
    Parameters:
    
    operation - operation
    
    path - path operated on (must not be null)
    
    exception - amazon exception raised
    
    Returns:
    
    an IOE which wraps the caught exception.
    
    See Also:
    
    S3 Error responses, Amazon S3 Error Best Practices, Dynamo DB Commmon errors
  - translateException
```
public static IOException translateException(@Nullable
                                             String operation,
                                             String path,
                                             com.amazonaws.SdkBaseException exception)
```
    Translate an exception raised in an operation into an IOException. The specific type of IOException depends on the class of AmazonClientException passed in, and any status codes included in the operation. That is: HTTP error codes are examined and can be used to build a more specific response.
    
    Parameters:
    
    operation - operation
    
    path - path operated on (may be null)
    
    exception - amazon exception raised
    
    Returns:
    
    an IOE which wraps the caught exception.
  - extractException
```
public static IOException extractException(String operation,
                                           String path,
                                           ExecutionException ee)
```
    Extract an exception from a failed future, and convert to an IOE.
    
    Parameters:
    
    operation - operation which failed
    
    path - path operated on (may be null)
    
    ee - execution exception
    
    Returns:
    
    an IOE which can be thrown
  - isThrottleException
```
public static boolean isThrottleException(Exception ex)
```
    Is the exception an instance of a throttling exception. That is an AmazonServiceException with a 503 response, any exception from DynamoDB for limits exceeded, an AWSServiceThrottledException, or anything which the AWS SDK's RetryUtils considers to be a throttling exception.
    
    Parameters:
    
    ex - exception to examine
    
    Returns:
    
    true if it is considered a throttling exception
  - isMessageTranslatableToEOF
```
public static boolean isMessageTranslatableToEOF(com.amazonaws.SdkBaseException ex)
```
    Cue that an AWS exception is likely to be an EOF Exception based on the message coming back from the client. This is likely to be brittle, so only a hint.
    
    Parameters:
    
    ex - exception
    
    Returns:
    
    true if this is believed to be a sign the connection was broken.
  - translateDynamoDBException
```
public static IOException translateDynamoDBException(String path,
                                                     String message,
                                                     com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException ddbException)
```
    Translate a DynamoDB exception into an IOException.
    
    Parameters:
    
    path - path in the DDB
    
    message - preformatted message for the exception
    
    ddbException - exception
    
    Returns:
    
    an exception to throw.
  - stringify
```
public static String stringify(com.amazonaws.AmazonServiceException e)
```
    Get low level details of an amazon exception for logging; multi-line.
    
    Parameters:
    
    e - exception
    
    Returns:
    
    string details
  - stringify
```
public static String stringify(com.amazonaws.services.s3.model.AmazonS3Exception e)
```
    Get low level details of an amazon exception for logging; multi-line.
    
    Parameters:
    
    e - exception
    
    Returns:
    
    string details
  - createFileStatus
```
public static S3AFileStatus createFileStatus(org.apache.hadoop.fs.Path keyPath,
                                             com.amazonaws.services.s3.model.S3ObjectSummary summary,
                                             long blockSize,
                                             String owner,
                                             String eTag,
                                             String versionId,
                                             boolean isCSEEnabled)
```
    Create a files status instance from a listing.
    
    Parameters:
    
    keyPath - path to entry
    
    summary - summary from AWS
    
    blockSize - block size to declare.
    
    owner - owner of the file
    
    eTag - S3 object eTag or null if unavailable
    
    versionId - S3 object versionId or null if unavailable
    
    isCSEEnabled - is client side encryption enabled?
    
    Returns:
    
    a status entry
  - createUploadFileStatus
```
public static S3AFileStatus createUploadFileStatus(org.apache.hadoop.fs.Path keyPath,
                                                   boolean isDir,
                                                   long size,
                                                   long blockSize,
                                                   String owner,
                                                   String eTag,
                                                   String versionId)
```
    Create a file status for object we just uploaded. For files, we use current time as modification time, since s3a uses S3's service-based modification time, which will not be available until we do a getFileStatus() later on.
    
    Parameters:
    
    keyPath - path for created object
    
    isDir - true iff directory
    
    size - file length
    
    blockSize - block size for file status
    
    owner - Hadoop username
    
    eTag - S3 object eTag or null if unavailable
    
    versionId - S3 object versionId or null if unavailable
    
    Returns:
    
    a status entry
  - objectRepresentsDirectory
```
public static boolean objectRepresentsDirectory(String name)
```
    Predicate: does the object represent a directory?.
    
    Parameters:
    
    name - object name
    
    Returns:
    
    true if it meets the criteria for being an object
  - dateToLong
```
public static long dateToLong(Date date)
```
    Date to long conversion. Handles null Dates that can be returned by AWS by returning 0
    
    Parameters:
    
    date - date from AWS query
    
    Returns:
    
    timestamp of the object
  - createAWSCredentialProviderSet
```
public static AWSCredentialProviderList createAWSCredentialProviderSet(@Nullable
                                                                       URI binding,
                                                                       org.apache.hadoop.conf.Configuration conf)
                                                                throws IOException
```
    Create the AWS credentials from the providers, the URI and the key Constants.AWS_CREDENTIALS_PROVIDER in the configuration.
    
    Parameters:
    
    binding - Binding URI -may be null
    
    conf - filesystem configuration
    
    Returns:
    
    a credentials provider list
    
    Throws:
    
    IOException - Problems loading the providers (including reading secrets from credential files).
  - loadAWSProviderClasses
```
public static List<Class<?>> loadAWSProviderClasses(org.apache.hadoop.conf.Configuration conf,
                                                    String key,
                                                    Class<?>... defaultValue)
                                             throws IOException
```
    Load list of AWS credential provider/credential provider factory classes.
    
    Parameters:
    
    conf - configuration
    
    key - key
    
    defaultValue - list of default values
    
    Returns:
    
    the list of classes, possibly empty
    
    Throws:
    
    IOException - on a failure to load the list.
  - buildAWSProviderList
```
public static AWSCredentialProviderList buildAWSProviderList(@Nullable
                                                             URI binding,
                                                             org.apache.hadoop.conf.Configuration conf,
                                                             String key,
                                                             List<Class<?>> defaultValues,
                                                             Set<Class<?>> forbidden)
                                                      throws IOException
```
    Load list of AWS credential provider/credential provider factory classes; support a forbidden list to prevent loops, mandate full secrets, etc.
    
    Parameters:
    
    binding - Binding URI -may be null
    
    conf - configuration
    
    key - key
    
    forbidden - a possibly empty set of forbidden classes.
    
    defaultValues - list of default providers.
    
    Returns:
    
    the list of classes, possibly empty
    
    Throws:
    
    IOException - on a failure to load the list.
  - setIfDefined
```
public static boolean setIfDefined(org.apache.hadoop.conf.Configuration config,
                                   String key,
                                   String val,
                                   String origin)
```
    Set a key if the value is non-empty.
    
    Parameters:
    
    config - config to patch
    
    key - key to set
    
    val - value to probe and set
    
    origin - origin
    
    Returns:
    
    true if the property was set
  - getAWSAccessKeys
```
public static S3xLoginHelper.Login getAWSAccessKeys(URI name,
                                                    org.apache.hadoop.conf.Configuration conf)
                                             throws IOException
```
    Return the access key and secret for S3 API use. or indicated in the UserInfo of the name URI param.
    
    Parameters:
    
    name - the URI for which we need the access keys; may be null
    
    conf - the Configuration object to interrogate for keys.
    
    Returns:
    
    AWSAccessKeys
    
    Throws:
    
    IOException - problems retrieving passwords from KMS.
  - lookupPassword
```
@Deprecated
public static String lookupPassword(String bucket,
                                                org.apache.hadoop.conf.Configuration conf,
                                                String baseKey,
                                                String overrideVal)
                                         throws IOException
```
    Deprecated.
    
    Get a password from a configuration, including JCEKS files, handling both the absolute key and bucket override.
    
    Parameters:
    
    bucket - bucket or "" if none known
    
    conf - configuration
    
    baseKey - base key to look up, e.g "fs.s3a.secret.key"
    
    overrideVal - override value: if non empty this is used instead of querying the configuration.
    
    Returns:
    
    a password or "".
    
    Throws:
    
    IOException - on any IO problem
    
    IllegalArgumentException - bad arguments
  - lookupPassword
```
public static String lookupPassword(String bucket,
                                    org.apache.hadoop.conf.Configuration conf,
                                    String baseKey)
                             throws IOException
```
    Get a password from a configuration, including JCEKS files, handling both the absolute key and bucket override.
    
    Parameters:
    
    bucket - bucket or "" if none known
    
    conf - configuration
    
    baseKey - base key to look up, e.g "fs.s3a.secret.key"
    
    Returns:
    
    a password or "".
    
    Throws:
    
    IOException - on any IO problem
    
    IllegalArgumentException - bad arguments
  - lookupPassword
```
public static String lookupPassword(String bucket,
                                    org.apache.hadoop.conf.Configuration conf,
                                    String baseKey,
                                    String overrideVal,
                                    String defVal)
                             throws IOException
```
    Get a password from a configuration, including JCEKS files, handling both the absolute key and bucket override.
    
    Parameters:
    
    bucket - bucket or "" if none known
    
    conf - configuration
    
    baseKey - base key to look up, e.g "fs.s3a.secret.key"
    
    overrideVal - override value: if non empty this is used instead of querying the configuration.
    
    defVal - value to return if there is no password
    
    Returns:
    
    a password or the value of defVal.
    
    Throws:
    
    IOException - on any IO problem
    
    IllegalArgumentException - bad arguments
  - stringify
```
public static String stringify(com.amazonaws.services.s3.model.S3ObjectSummary summary)
```
    String information about a summary entry for debug messages.
    
    Parameters:
    
    summary - summary object
    
    Returns:
    
    string value
  - intOption
```
public static int intOption(org.apache.hadoop.conf.Configuration conf,
                            String key,
                            int defVal,
                            int min)
```
    Get a integer option >= the minimum allowed value.
    
    Parameters:
    
    conf - configuration
    
    key - key to look up
    
    defVal - default value
    
    min - minimum value
    
    Returns:
    
    the value
    
    Throws:
    
    IllegalArgumentException - if the value is below the minimum
  - longOption
```
public static long longOption(org.apache.hadoop.conf.Configuration conf,
                              String key,
                              long defVal,
                              long min)
```
    Get a long option >= the minimum allowed value.
    
    Parameters:
    
    conf - configuration
    
    key - key to look up
    
    defVal - default value
    
    min - minimum value
    
    Returns:
    
    the value
    
    Throws:
    
    IllegalArgumentException - if the value is below the minimum
  - longBytesOption
```
public static long longBytesOption(org.apache.hadoop.conf.Configuration conf,
                                   String key,
                                   long defVal,
                                   long min)
```
    Get a long option >= the minimum allowed value, supporting memory prefixes K,M,G,T,P.
    
    Parameters:
    
    conf - configuration
    
    key - key to look up
    
    defVal - default value
    
    min - minimum value
    
    Returns:
    
    the value
    
    Throws:
    
    IllegalArgumentException - if the value is below the minimum
  - getMultipartSizeProperty
```
public static long getMultipartSizeProperty(org.apache.hadoop.conf.Configuration conf,
                                            String property,
                                            long defVal)
```
    Get a size property from the configuration: this property must be at least equal to Constants.MULTIPART_MIN_SIZE. If it is too small, it is rounded up to that minimum, and a warning printed.
    
    Parameters:
    
    conf - configuration
    
    property - property name
    
    defVal - default value
    
    Returns:
    
    the value, guaranteed to be above the minimum size
  - ensureOutputParameterInRange
```
public static int ensureOutputParameterInRange(String name,
                                               long size)
```
    Ensure that the long value is in the range of an integer.
    
    Parameters:
    
    name - property name for error messages
    
    size - original size
    
    Returns:
    
    the size, guaranteed to be less than or equal to the max value of an integer.
  - propagateBucketOptions
```
public static org.apache.hadoop.conf.Configuration propagateBucketOptions(org.apache.hadoop.conf.Configuration source,
                                                                          String bucket)
```
    Propagates bucket-specific settings into generic S3A configuration keys. This is done by propagating the values of the form fs.s3a.bucket.${bucket}.key to fs.s3a.key, for all values of "key" other than a small set of unmodifiable values. The source of the updated property is set to the key name of the bucket property, to aid in diagnostics of where things came from. Returns a new configuration. Why the clone? You can use the same conf for different filesystems, and the original values are not updated. The fs.s3a.impl property cannot be set, nor can any with the prefix fs.s3a.bucket. This method does not propagate security provider path information from the S3A property into the Hadoop common provider: callers must call patchSecurityCredentialProviders(Configuration) explicitly.
    
    Parameters:
    
    source - Source Configuration object.
    
    bucket - bucket name. Must not be empty.
    
    Returns:
    
    a (potentially) patched clone of the original.
  - deleteQuietly
```
public static void deleteQuietly(org.apache.hadoop.fs.FileSystem fs,
                                 org.apache.hadoop.fs.Path path,
                                 boolean recursive)
```
    Delete a path quietly: failures are logged at DEBUG.
    
    Parameters:
    
    fs - filesystem
    
    path - path
    
    recursive - recursive?
  - deleteWithWarning
```
public static void deleteWithWarning(org.apache.hadoop.fs.FileSystem fs,
                                     org.apache.hadoop.fs.Path path,
                                     boolean recursive)
```
    Delete a path: failures are logged at WARN.
    
    Parameters:
    
    fs - filesystem
    
    path - path
    
    recursive - recursive?
  - createAwsConf
```
@Deprecated
public static com.amazonaws.ClientConfiguration createAwsConf(org.apache.hadoop.conf.Configuration conf,
                                                                          String bucket)
                                                                   throws IOException
```
    Deprecated. use createAwsConf(Configuration, String, String)
    
    Create a new AWS ClientConfiguration. All clients to AWS services MUST use this for consistent setup of connectivity, UA, proxy settings.
    
    Parameters:
    
    conf - The Hadoop configuration
    
    bucket - Optional bucket to use to look up per-bucket proxy secrets
    
    Returns:
    
    new AWS client configuration
    
    Throws:
    
    IOException - problem creating AWS client configuration
  - createAwsConf
```
public static com.amazonaws.ClientConfiguration createAwsConf(org.apache.hadoop.conf.Configuration conf,
                                                              String bucket,
                                                              String awsServiceIdentifier)
                                                       throws IOException
```
    Create a new AWS ClientConfiguration. All clients to AWS services MUST use this or the equivalents for the specific service for consistent setup of connectivity, UA, proxy settings.
    
    Parameters:
    
    conf - The Hadoop configuration
    
    bucket - Optional bucket to use to look up per-bucket proxy secrets
    
    awsServiceIdentifier - a string representing the AWS service (S3, DDB, etc) for which the ClientConfiguration is being created.
    
    Returns:
    
    new AWS client configuration
    
    Throws:
    
    IOException - problem creating AWS client configuration
  - initConnectionSettings
```
public static void initConnectionSettings(org.apache.hadoop.conf.Configuration conf,
                                          com.amazonaws.ClientConfiguration awsConf)
                                   throws IOException
```
    Initializes all AWS SDK settings related to connection management.
    
    Parameters:
    
    conf - Hadoop configuration
    
    awsConf - AWS SDK configuration
    
    Throws:
    
    IOException - if there was an error initializing the protocol settings
  - initProxySupport
```
public static void initProxySupport(org.apache.hadoop.conf.Configuration conf,
                                    String bucket,
                                    com.amazonaws.ClientConfiguration awsConf)
                             throws IllegalArgumentException,
                                    IOException
```
    Initializes AWS SDK proxy support in the AWS client configuration if the S3A settings enable it.
    
    Parameters:
    
    conf - Hadoop configuration
    
    bucket - Optional bucket to use to look up per-bucket proxy secrets
    
    awsConf - AWS SDK configuration to update
    
    Throws:
    
    IllegalArgumentException - if misconfigured
    
    IOException - problem getting username/secret from password source.
  - iteratorToStatuses
```
public static S3AFileStatus[] iteratorToStatuses(org.apache.hadoop.fs.RemoteIterator<S3AFileStatus> iterator,
                                                 Set<org.apache.hadoop.fs.Path> tombstones)
                                          throws IOException
```
    Convert the data of an iterator of S3AFileStatus to an array. Given tombstones are filtered out. If the iterator does return any item, an empty array is returned.
    
    Parameters:
    
    iterator - a non-null iterator
    
    tombstones - possibly empty set of tombstones
    
    Returns:
    
    a possibly-empty array of file status entries
    
    Throws:
    
    IOException - failure
  - applyLocatedFiles
```
public static long applyLocatedFiles(org.apache.hadoop.fs.RemoteIterator<? extends org.apache.hadoop.fs.LocatedFileStatus> iterator,
                                     S3AUtils.CallOnLocatedFileStatus eval)
                              throws IOException
```
    Apply an operation to every LocatedFileStatus in a remote iterator.
    
    Parameters:
    
    iterator - iterator from a list
    
    eval - closure to evaluate
    
    Returns:
    
    the number of files processed
    
    Throws:
    
    IOException - anything in the closure, or iteration logic.
  - mapLocatedFiles
```
public static <T> List<T> mapLocatedFiles(org.apache.hadoop.fs.RemoteIterator<? extends org.apache.hadoop.fs.LocatedFileStatus> iterator,
                                          S3AUtils.LocatedFileStatusMap<T> eval)
                                   throws IOException
```
    Map an operation to every LocatedFileStatus in a remote iterator, returning a list of the results.
    
    Type Parameters:
    
    T - return type of map
    
    Parameters:
    
    iterator - iterator from a list
    
    eval - closure to evaluate
    
    Returns:
    
    the list of mapped results.
    
    Throws:
    
    IOException - anything in the closure, or iteration logic.
  - flatmapLocatedFiles
```
public static <T> List<T> flatmapLocatedFiles(org.apache.hadoop.fs.RemoteIterator<org.apache.hadoop.fs.LocatedFileStatus> iterator,
                                              S3AUtils.LocatedFileStatusMap<Optional<T>> eval)
                                       throws IOException
```
    Map an operation to every LocatedFileStatus in a remote iterator, returning a list of the all results which were not empty.
    
    Type Parameters:
    
    T - return type of map
    
    Parameters:
    
    iterator - iterator from a list
    
    eval - closure to evaluate
    
    Returns:
    
    the flattened list of mapped results.
    
    Throws:
    
    IOException - anything in the closure, or iteration logic.
  - listAndFilter
```
public static List<org.apache.hadoop.fs.LocatedFileStatus> listAndFilter(org.apache.hadoop.fs.FileSystem fileSystem,
                                                                         org.apache.hadoop.fs.Path path,
                                                                         boolean recursive,
                                                                         org.apache.hadoop.fs.PathFilter filter)
                                                                  throws IOException
```
    List located files and filter them as a classic listFiles(path, filter) would do.
    
    Parameters:
    
    fileSystem - filesystem
    
    path - path to list
    
    recursive - recursive listing?
    
    filter - filter for the filename
    
    Returns:
    
    the filtered list of entries
    
    Throws:
    
    IOException - IO failure.
  - maybe
```
public static <T> Optional<T> maybe(boolean include,
                                    T value)
```
    Convert a value into a non-empty Optional instance if the value of include is true.
    
    Type Parameters:
    
    T - type of option.
    
    Parameters:
    
    include - flag to indicate the value is to be included.
    
    value - value to return
    
    Returns:
    
    if include is false, Optional.empty. Otherwise, the value.
  - getS3EncryptionKey
```
public static String getS3EncryptionKey(String bucket,
                                        org.apache.hadoop.conf.Configuration conf)
```
    Get any S3 encryption key, without propagating exceptions from JCEKs files.
    
    Parameters:
    
    bucket - bucket to query for
    
    conf - configuration to examine
    
    Returns:
    
    the encryption key or ""
    
    Throws:
    
    IllegalArgumentException - bad arguments.
  - getS3EncryptionKey
```
public static String getS3EncryptionKey(String bucket,
                                        org.apache.hadoop.conf.Configuration conf,
                                        boolean propagateExceptions)
                                 throws IOException
```
    Get any SSE/CSE key from a configuration/credential provider. This operation handles the case where the option has been set in the provider or configuration to the option SERVER_SIDE_ENCRYPTION_KEY. IOExceptions raised during retrieval are swallowed.
    
    Parameters:
    
    bucket - bucket to query for
    
    conf - configuration to examine
    
    propagateExceptions - should IO exceptions be rethrown?
    
    Returns:
    
    the encryption key or ""
    
    Throws:
    
    IllegalArgumentException - bad arguments.
    
    IOException - if propagateExceptions==true and reading a JCEKS file raised an IOE
  - getEncryptionAlgorithm
```
public static S3AEncryptionMethods getEncryptionAlgorithm(String bucket,
                                                          org.apache.hadoop.conf.Configuration conf)
                                                   throws IOException
```
    Get the server-side encryption or client side encryption algorithm. This includes validation of the configuration, checking the state of the encryption key given the chosen algorithm.
    
    Parameters:
    
    bucket - bucket to query for
    
    conf - configuration to scan
    
    Returns:
    
    the encryption mechanism (which will be NONE unless one is set.
    
    Throws:
    
    IOException - on JCKES lookup or invalid method/key configuration.
  - buildEncryptionSecrets
```
public static EncryptionSecrets buildEncryptionSecrets(String bucket,
                                                       org.apache.hadoop.conf.Configuration conf)
                                                throws IOException
```
    Get the server-side encryption or client side encryption algorithm. This includes validation of the configuration, checking the state of the encryption key given the chosen algorithm.
    
    Parameters:
    
    bucket - bucket to query for
    
    conf - configuration to scan
    
    Returns:
    
    the encryption mechanism (which will be NONE unless one is set and secrets.
    
    Throws:
    
    IOException - on JCKES lookup or invalid method/key configuration.
  - closeAll
```
@Deprecated
public static void closeAll(org.slf4j.Logger log,
                                        Closeable... closeables)
```
    Deprecated.
    
    Close the Closeable objects and ignore any Exception or null pointers. This is obsolete: use IOUtils.cleanupWithLogger(Logger, Closeable...)
    
    Parameters:
    
    log - the log to log at debug level. Can be null.
    
    closeables - the objects to close
  - closeAutocloseables
```
public static void closeAutocloseables(org.slf4j.Logger log,
                                       AutoCloseable... closeables)
```
    Close the Closeable objects and ignore any Exception or null pointers. (This is the SLF4J equivalent of that in IOUtils).
    
    Parameters:
    
    log - the log to log at debug level. Can be null.
    
    closeables - the objects to close
  - setBucketOption
```
public static void setBucketOption(org.apache.hadoop.conf.Configuration conf,
                                   String bucket,
                                   String genericKey,
                                   String value)
```
    Set a bucket-specific property to a particular value. If the generic key passed in has an fs.s3a. prefix, that's stripped off, so that when the the bucket properties are propagated down to the generic values, that value gets copied down.
    
    Parameters:
    
    conf - configuration to set
    
    bucket - bucket name
    
    genericKey - key; can start with "fs.s3a."
    
    value - value to set
  - clearBucketOption
```
public static void clearBucketOption(org.apache.hadoop.conf.Configuration conf,
                                     String bucket,
                                     String genericKey)
```
    Clear a bucket-specific property. If the generic key passed in has an fs.s3a. prefix, that's stripped off, so that when the the bucket properties are propagated down to the generic values, that value gets copied down.
    
    Parameters:
    
    conf - configuration to set
    
    bucket - bucket name
    
    genericKey - key; can start with "fs.s3a."
  - getBucketOption
```
public static String getBucketOption(org.apache.hadoop.conf.Configuration conf,
                                     String bucket,
                                     String genericKey)
```
    Get a bucket-specific property. If the generic key passed in has an fs.s3a. prefix, that's stripped off.
    
    Parameters:
    
    conf - configuration to set
    
    bucket - bucket name
    
    genericKey - key; can start with "fs.s3a."
    
    Returns:
    
    the bucket option, null if there is none
  - maybeAddTrailingSlash
```
public static String maybeAddTrailingSlash(String key)
```
    Turns a path (relative or otherwise) into an S3 key, adding a trailing "/" if the path is not the root and does not already have a "/" at the end.
    
    Parameters:
    
    key - s3 key or ""
    
    Returns:
    
    the with a trailing "/", or, if it is the root key, "",

Class S3AUtils

Nested Class Summary

Field Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

SSE_C_NO_KEY_ERROR

SSE_S3_WITH_KEY_ERROR

EOF_MESSAGE_IN_XML_PARSER

EOF_READ_DIFFERENT_LENGTH

E_FORBIDDEN_AWS_PROVIDER

STANDARD_AWS_PROVIDERS

HIDDEN_FILE_FILTER

ACCEPT_ALL

Method Detail

translateException

translateException

extractException

isThrottleException

isMessageTranslatableToEOF

translateDynamoDBException

stringify

stringify

createFileStatus

createUploadFileStatus

objectRepresentsDirectory

dateToLong

createAWSCredentialProviderSet

loadAWSProviderClasses

buildAWSProviderList

setIfDefined

getAWSAccessKeys

lookupPassword

lookupPassword

lookupPassword

stringify

intOption

longOption

longBytesOption

getMultipartSizeProperty

ensureOutputParameterInRange

propagateBucketOptions

deleteQuietly

deleteWithWarning

createAwsConf

createAwsConf

initConnectionSettings

initProxySupport

iteratorToStatuses

applyLocatedFiles

mapLocatedFiles

flatmapLocatedFiles

listAndFilter

maybe

getS3EncryptionKey

getS3EncryptionKey

getEncryptionAlgorithm

buildEncryptionSecrets

closeAll

closeAutocloseables

setBucketOption

clearBucketOption

getBucketOption

maybeAddTrailingSlash