org.apache.hadoop.fs.s3a

Class DefaultS3ClientFactory

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.hadoop.fs.s3a.DefaultS3ClientFactory

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, S3ClientFactory

Direct Known Subclasses:

InconsistentS3ClientFactory

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public class DefaultS3ClientFactory
extends org.apache.hadoop.conf.Configured
implements S3ClientFactory

The default S3ClientFactory implementation. This calls the AWS SDK to configure and create an AmazonS3Client that communicates with the S3 service.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.fs.s3a.S3ClientFactory

S3ClientFactory.S3ClientCreationParameters

Field Summary

Fields

Modifier and Type	Field and Description
protected static org.slf4j.Logger	LOG Subclasses refer to this.

Constructor Summary

Constructors

Constructor and Description
DefaultS3ClientFactory()

Method Summary

Modifier and Type	Method and Description
protected com.amazonaws.services.s3.AmazonS3	buildAmazonS3Client(com.amazonaws.ClientConfiguration awsConf, S3ClientFactory.S3ClientCreationParameters parameters) Use the Builder API to create an AWS S3 client.
protected com.amazonaws.services.s3.AmazonS3	buildAmazonS3EncryptionClient(com.amazonaws.ClientConfiguration awsConf, S3ClientFactory.S3ClientCreationParameters parameters) Create an AmazonS3 client of type AmazonS3EncryptionV2 if CSE is enabled.
protected static com.amazonaws.services.s3.AmazonS3	configureAmazonS3Client(com.amazonaws.services.s3.AmazonS3 s3, String endPoint, boolean pathStyleAccess) Configure classic S3 client.
static com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration	createEndpointConfiguration(String endpoint, com.amazonaws.ClientConfiguration awsConf, String awsRegion) Given an endpoint string, return an endpoint config, or null, if none is needed.
com.amazonaws.services.s3.AmazonS3	createS3Client(URI uri, S3ClientFactory.S3ClientCreationParameters parameters) Create the client by preparing the AwsConf configuration and then invoking buildAmazonS3Client().

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

protected static final org.slf4j.Logger LOG

Subclasses refer to this.

Constructor Detail

DefaultS3ClientFactory

public DefaultS3ClientFactory()

Method Detail

createS3Client

public com.amazonaws.services.s3.AmazonS3 createS3Client(URI uri,
                                                         S3ClientFactory.S3ClientCreationParameters parameters)
                                                  throws IOException

Create the client by preparing the AwsConf configuration and then invoking buildAmazonS3Client().

Specified by:

createS3Client in interface S3ClientFactory

Parameters:

uri - S3A file system URI

parameters - parameter object

Returns:

S3 client

Throws:

IOException - IO problem

buildAmazonS3EncryptionClient

protected com.amazonaws.services.s3.AmazonS3 buildAmazonS3EncryptionClient(com.amazonaws.ClientConfiguration awsConf,
                                                                           S3ClientFactory.S3ClientCreationParameters parameters)
                                                                    throws IOException

Create an AmazonS3 client of type AmazonS3EncryptionV2 if CSE is enabled.

Parameters:

awsConf - AWS configuration.

parameters - parameters.

Returns:

new AmazonS3 client.

Throws:

IOException - if lookupPassword() has any problem.

buildAmazonS3Client

protected com.amazonaws.services.s3.AmazonS3 buildAmazonS3Client(com.amazonaws.ClientConfiguration awsConf,
                                                                 S3ClientFactory.S3ClientCreationParameters parameters)

Use the Builder API to create an AWS S3 client.

This has a more complex endpoint configuration mechanism which initially caused problems; the withForceGlobalBucketAccessEnabled(true) command is critical here.

Parameters:

awsConf - AWS configuration

parameters - parameters

Returns:

new AmazonS3 client

Throws:

com.amazonaws.SdkClientException - if the configuration is invalid.

configureAmazonS3Client

protected static com.amazonaws.services.s3.AmazonS3 configureAmazonS3Client(com.amazonaws.services.s3.AmazonS3 s3,
                                                                            String endPoint,
                                                                            boolean pathStyleAccess)
                                                                     throws IllegalArgumentException

Configure classic S3 client.

This includes: endpoint, Path Access and possibly other options.

Parameters:

s3 - S3 Client.

endPoint - s3 endpoint, may be empty

pathStyleAccess - enable path style access?

Returns:

S3 client

Throws:

IllegalArgumentException - if misconfigured

createEndpointConfiguration

public static com.amazonaws.client.builder.AwsClientBuilder.EndpointConfiguration createEndpointConfiguration(String endpoint,
                                                                                                              com.amazonaws.ClientConfiguration awsConf,
                                                                                                              String awsRegion)

Given an endpoint string, return an endpoint config, or null, if none is needed.

This is a pretty painful piece of code. It is trying to replicate what AwsClient.setEndpoint() does, because you can't call that setter on an AwsClient constructed via the builder, and you can't pass a metrics collector down except through the builder.

Note also that AWS signing is a mystery which nobody fully understands, especially given all problems surface in a "400 bad request" response, which, like all security systems, provides minimal diagnostics out of fear of leaking secrets.

Parameters:

endpoint - possibly null endpoint.

awsConf - config to build the URI from.

awsRegion - AWS S3 Region if the corresponding config is set.

Returns:

a configuration for the S3 client builder.