org.apache.hadoop.fs.s3a

Class DefaultS3ClientFactory

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.s3a.DefaultS3ClientFactory

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, S3ClientFactory

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public class DefaultS3ClientFactory
extends org.apache.hadoop.conf.Configured
implements S3ClientFactory

The default S3ClientFactory implementation. This calls the AWS SDK to configure and create an AmazonS3Client that communicates with the S3 service.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.fs.s3a.S3ClientFactory

S3ClientFactory.S3ClientCreationParameters

Field Summary

Fields

Modifier and Type	Field and Description
static String	ERROR_ENDPOINT_WITH_FIPS Error message when an endpoint is set with FIPS enabled: "An endpoint cannot set when fs.s3a.endpoint.fips is true".
protected static org.slf4j.Logger	LOG Subclasses refer to this.

Constructor Summary

Constructors

Constructor and Description
DefaultS3ClientFactory()

Method Summary

Modifier and Type	Method and Description
protected software.amazon.awssdk.core.client.config.ClientOverrideConfiguration.Builder	createClientOverrideConfiguration(S3ClientFactory.S3ClientCreationParameters parameters, org.apache.hadoop.conf.Configuration conf) Create an override configuration for an S3 client.
software.amazon.awssdk.services.s3.S3AsyncClient	createS3AsyncClient(URI uri, S3ClientFactory.S3ClientCreationParameters parameters) Creates a new S3AsyncClient.
software.amazon.awssdk.services.s3.S3Client	createS3Client(URI uri, S3ClientFactory.S3ClientCreationParameters parameters) Creates a new S3Client.
software.amazon.awssdk.transfer.s3.S3TransferManager	createS3TransferManager(software.amazon.awssdk.services.s3.S3AsyncClient s3AsyncClient) Creates a new S3TransferManager.

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

protected static final org.slf4j.Logger LOG

Subclasses refer to this.

ERROR_ENDPOINT_WITH_FIPS

@VisibleForTesting
public static final String ERROR_ENDPOINT_WITH_FIPS

Error message when an endpoint is set with FIPS enabled: "An endpoint cannot set when fs.s3a.endpoint.fips is true".

See Also:

Constant Field Values

Constructor Detail

DefaultS3ClientFactory

public DefaultS3ClientFactory()

Method Detail

createS3Client

public software.amazon.awssdk.services.s3.S3Client createS3Client(URI uri,
                                                                  S3ClientFactory.S3ClientCreationParameters parameters)
                                                           throws IOException

Description copied from interface: S3ClientFactory

Creates a new S3Client. The client returned supports synchronous operations. For asynchronous operations, use S3ClientFactory.createS3AsyncClient(URI, S3ClientCreationParameters).

Specified by:

createS3Client in interface S3ClientFactory

Parameters:

uri - S3A file system URI

parameters - parameter object

Returns:

S3 client

Throws:

IOException - on any IO problem

createS3AsyncClient

public software.amazon.awssdk.services.s3.S3AsyncClient createS3AsyncClient(URI uri,
                                                                            S3ClientFactory.S3ClientCreationParameters parameters)
                                                                     throws IOException

Description copied from interface: S3ClientFactory

Creates a new S3AsyncClient. The client returned supports asynchronous operations. For synchronous operations, use S3ClientFactory.createS3Client(URI, S3ClientCreationParameters).

Specified by:

createS3AsyncClient in interface S3ClientFactory

Parameters:

uri - S3A file system URI

parameters - parameter object

Returns:

Async S3 client

Throws:

IOException - on any IO problem

createS3TransferManager

public software.amazon.awssdk.transfer.s3.S3TransferManager createS3TransferManager(software.amazon.awssdk.services.s3.S3AsyncClient s3AsyncClient)

Description copied from interface: S3ClientFactory

Creates a new S3TransferManager.

Specified by:

createS3TransferManager in interface S3ClientFactory

Parameters:

s3AsyncClient - the async client to be used by the TM.

Returns:

S3 transfer manager

createClientOverrideConfiguration

protected software.amazon.awssdk.core.client.config.ClientOverrideConfiguration.Builder createClientOverrideConfiguration(S3ClientFactory.S3ClientCreationParameters parameters,
                                                                                                                          org.apache.hadoop.conf.Configuration conf)
                                                                                                                   throws IOException

Create an override configuration for an S3 client.

Parameters:

parameters - parameter object

conf - configuration object

Returns:

the override configuration

Throws:

IOException - any IOE raised, or translated exception

RuntimeException - some failures creating an http signer

IOException - any IOE raised, or translated exception