org.apache.hadoop.fs.s3a.impl

Class RequestFactoryImpl

java.lang.Object

org.apache.hadoop.fs.s3a.impl.RequestFactoryImpl

All Implemented Interfaces:

RequestFactory

public class RequestFactoryImpl
extends Object
implements RequestFactory

The standard implementation of the request factory. This creates AWS SDK request classes for the specific bucket, with standard options/headers set. It is also where custom setting parameters can take place. All creation of AWS S3 requests MUST be through this class so that common options (encryption etc.) can be added here, and so that any chained transformation of requests can be applied. This is where audit span information is added to the requests, until it is done in the AWS SDK itself. All created requests will be passed through RequestFactoryImpl.PrepareRequest.prepareRequest(AmazonWebServiceRequest) before being returned to the caller.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	RequestFactoryImpl.PrepareRequest This is a callback for anything to "prepare" every request after creation.
static class	RequestFactoryImpl.RequestFactoryBuilder Builder.

Field Summary

Fields

Modifier and Type	Field and Description
static org.slf4j.Logger	LOG

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	RequestFactoryImpl(RequestFactoryImpl.RequestFactoryBuilder builder) Constructor.

Method Summary

Modifier and Type	Method and Description
static RequestFactoryImpl.RequestFactoryBuilder	builder() Create a builder.
protected void	copyEncryptionParameters(com.amazonaws.services.s3.model.ObjectMetadata srcom, com.amazonaws.services.s3.model.CopyObjectRequest copyObjectRequest) Propagate encryption parameters from source file if set else use the current filesystem encryption settings.
Optional<com.amazonaws.services.s3.model.SSEAwsKeyManagementParams>	generateSSEAwsKeyParams() Create the AWS SDK structure used to configure SSE, if the encryption secrets contain the information/settings for this.
Optional<com.amazonaws.services.s3.model.SSECustomerKey>	generateSSECustomerKey() Create the SSE-C structure for the AWS SDK, if the encryption secrets contain the information/settings for this.
protected String	getBucket() Get the target bucket.
com.amazonaws.services.s3.model.CannedAccessControlList	getCannedACL() Get the canned ACL of this FS.
S3AEncryptionMethods	getServerSideEncryptionAlgorithm() Get the encryption algorithm of this endpoint.
com.amazonaws.services.s3.model.AbortMultipartUploadRequest	newAbortMultipartUploadRequest(String destKey, String uploadId) Abort a multipart upload.
com.amazonaws.services.s3.model.DeleteObjectsRequest	newBulkDeleteRequest(List<com.amazonaws.services.s3.model.DeleteObjectsRequest.KeyVersion> keysToDelete, boolean quiet) Bulk delete request.
com.amazonaws.services.s3.model.CompleteMultipartUploadRequest	newCompleteMultipartUploadRequest(String destKey, String uploadId, List<com.amazonaws.services.s3.model.PartETag> partETags) Complete a multipart upload.
com.amazonaws.services.s3.model.CopyObjectRequest	newCopyObjectRequest(String srcKey, String dstKey, com.amazonaws.services.s3.model.ObjectMetadata srcom) Create a copy request.
com.amazonaws.services.s3.model.DeleteObjectRequest	newDeleteObjectRequest(String key) Create a request to delete a single object.
com.amazonaws.services.s3.model.PutObjectRequest	newDirectoryMarkerRequest(String directory) Create a PutObjectRequest request for creating an empty directory.
com.amazonaws.services.s3.model.GetObjectMetadataRequest	newGetObjectMetadataRequest(String key) Create a HEAD request.
com.amazonaws.services.s3.model.GetObjectRequest	newGetObjectRequest(String key) Create a GET request.
com.amazonaws.services.s3.model.ListMultipartUploadsRequest	newListMultipartUploadsRequest(String prefix) List all multipart uploads under a prefix.
com.amazonaws.services.s3.model.ListNextBatchOfObjectsRequest	newListNextBatchOfObjectsRequest(com.amazonaws.services.s3.model.ObjectListing prev) Create the next V1 page list request, following on from the previous response.
com.amazonaws.services.s3.model.ListObjectsRequest	newListObjectsV1Request(String key, String delimiter, int maxKeys) Create the (legacy) V1 list request.
com.amazonaws.services.s3.model.ListObjectsV2Request	newListObjectsV2Request(String key, String delimiter, int maxKeys) Create a V2 list request.
com.amazonaws.services.s3.model.InitiateMultipartUploadRequest	newMultipartUploadRequest(String destKey) Start a multipart upload.
com.amazonaws.services.s3.model.ObjectMetadata	newObjectMetadata(long length) Create a new object metadata instance.
com.amazonaws.services.s3.model.PutObjectRequest	newPutObjectRequest(String key, com.amazonaws.services.s3.model.ObjectMetadata metadata, File srcfile) Create a putObject request.
com.amazonaws.services.s3.model.PutObjectRequest	newPutObjectRequest(String key, com.amazonaws.services.s3.model.ObjectMetadata metadata, InputStream inputStream) Create a PutObjectRequest request.
com.amazonaws.services.s3.model.SelectObjectContentRequest	newSelectRequest(String key) Create a S3 Select request for the destination object.
com.amazonaws.services.s3.model.UploadPartRequest	newUploadPartRequest(String destKey, String uploadId, int partNumber, int size, InputStream uploadStream, File sourceFile, long offset) Create and initialize a part request of a multipart upload.
void	setEncryptionSecrets(EncryptionSecrets secrets) Set the encryption secrets for all subsequent requests.
protected void	setOptionalGetObjectMetadataParameters(com.amazonaws.services.s3.model.GetObjectMetadataRequest request) Sets server side encryption parameters to the GET reuquest.
protected void	setOptionalMultipartUploadRequestParameters(com.amazonaws.services.s3.model.InitiateMultipartUploadRequest request) Set the optional parameters when initiating the request (encryption, headers, storage, etc).
protected void	setOptionalObjectMetadata(com.amazonaws.services.s3.model.ObjectMetadata metadata) Set the optional metadata for an object being created or copied.
protected void	setOptionalPutRequestParameters(com.amazonaws.services.s3.model.PutObjectRequest request) Set the optional parameters for a PUT request.
protected void	setOptionalUploadPartRequestParameters(com.amazonaws.services.s3.model.UploadPartRequest request) Sets server side encryption parameters to the part upload request when encryption is enabled.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

Constructor Detail

RequestFactoryImpl

protected RequestFactoryImpl(RequestFactoryImpl.RequestFactoryBuilder builder)

Constructor.

Parameters:

builder - builder with all the configuration.

Method Detail

getCannedACL

public com.amazonaws.services.s3.model.CannedAccessControlList getCannedACL()

Get the canned ACL of this FS.

Specified by:

getCannedACL in interface RequestFactory

Returns:

an ACL, if any

getBucket

protected String getBucket()

Get the target bucket.

Returns:

the bucket.

generateSSEAwsKeyParams

public Optional<com.amazonaws.services.s3.model.SSEAwsKeyManagementParams> generateSSEAwsKeyParams()

Create the AWS SDK structure used to configure SSE, if the encryption secrets contain the information/settings for this.

Specified by:

generateSSEAwsKeyParams in interface RequestFactory

Returns:

an optional set of KMS Key settings

generateSSECustomerKey

public Optional<com.amazonaws.services.s3.model.SSECustomerKey> generateSSECustomerKey()

Create the SSE-C structure for the AWS SDK, if the encryption secrets contain the information/settings for this. This will contain a secret extracted from the bucket/configuration.

Specified by:

generateSSECustomerKey in interface RequestFactory

Returns:

an optional customer key.

getServerSideEncryptionAlgorithm

public S3AEncryptionMethods getServerSideEncryptionAlgorithm()

Get the encryption algorithm of this endpoint.

Specified by:

getServerSideEncryptionAlgorithm in interface RequestFactory

Returns:

the encryption algorithm.

setOptionalUploadPartRequestParameters

protected void setOptionalUploadPartRequestParameters(com.amazonaws.services.s3.model.UploadPartRequest request)

Sets server side encryption parameters to the part upload request when encryption is enabled.

Parameters:

request - upload part request

setOptionalGetObjectMetadataParameters

protected void setOptionalGetObjectMetadataParameters(com.amazonaws.services.s3.model.GetObjectMetadataRequest request)

Sets server side encryption parameters to the GET reuquest. request when encryption is enabled.

Parameters:

request - upload part request

setOptionalMultipartUploadRequestParameters

protected void setOptionalMultipartUploadRequestParameters(com.amazonaws.services.s3.model.InitiateMultipartUploadRequest request)

Set the optional parameters when initiating the request (encryption, headers, storage, etc).

Parameters:

request - request to patch.

setOptionalPutRequestParameters

protected void setOptionalPutRequestParameters(com.amazonaws.services.s3.model.PutObjectRequest request)

Set the optional parameters for a PUT request.

Parameters:

request - request to patch.

setOptionalObjectMetadata

protected void setOptionalObjectMetadata(com.amazonaws.services.s3.model.ObjectMetadata metadata)

Set the optional metadata for an object being created or copied.

Parameters:

metadata - to update.

newObjectMetadata

public com.amazonaws.services.s3.model.ObjectMetadata newObjectMetadata(long length)

Create a new object metadata instance. Any standard metadata headers are added here, for example: encryption.

Specified by:

newObjectMetadata in interface RequestFactory

Parameters:

length - length of data to set in header; Ignored if negative

Returns:

a new metadata instance

newCopyObjectRequest

public com.amazonaws.services.s3.model.CopyObjectRequest newCopyObjectRequest(String srcKey,
                                                                              String dstKey,
                                                                              com.amazonaws.services.s3.model.ObjectMetadata srcom)

Description copied from interface: RequestFactory

Create a copy request. This includes the work of copying the relevant parts of the metadata from the source

Specified by:

newCopyObjectRequest in interface RequestFactory

Parameters:

srcKey - source

dstKey - destination

srcom - source object metadata.

Returns:

the request

copyEncryptionParameters

protected void copyEncryptionParameters(com.amazonaws.services.s3.model.ObjectMetadata srcom,
                                        com.amazonaws.services.s3.model.CopyObjectRequest copyObjectRequest)

Propagate encryption parameters from source file if set else use the current filesystem encryption settings.

Parameters:

srcom - source object metadata.

copyObjectRequest - copy object request body.

newPutObjectRequest

public com.amazonaws.services.s3.model.PutObjectRequest newPutObjectRequest(String key,
                                                                            com.amazonaws.services.s3.model.ObjectMetadata metadata,
                                                                            File srcfile)

Create a putObject request. Adds the ACL and metadata

Specified by:

newPutObjectRequest in interface RequestFactory

Parameters:

key - key of object

metadata - metadata header

srcfile - source file

Returns:

the request

newPutObjectRequest

public com.amazonaws.services.s3.model.PutObjectRequest newPutObjectRequest(String key,
                                                                            com.amazonaws.services.s3.model.ObjectMetadata metadata,
                                                                            InputStream inputStream)

Create a PutObjectRequest request. The metadata is assumed to have been configured with the size of the operation.

Specified by:

newPutObjectRequest in interface RequestFactory

Parameters:

key - key of object

metadata - metadata header

inputStream - source data.

Returns:

the request

newDirectoryMarkerRequest

public com.amazonaws.services.s3.model.PutObjectRequest newDirectoryMarkerRequest(String directory)

Description copied from interface: RequestFactory

Create a PutObjectRequest request for creating an empty directory.

Specified by:

newDirectoryMarkerRequest in interface RequestFactory

Parameters:

directory - destination directory.

Returns:

request for a zero byte upload.

newListMultipartUploadsRequest

public com.amazonaws.services.s3.model.ListMultipartUploadsRequest newListMultipartUploadsRequest(String prefix)

Description copied from interface: RequestFactory

List all multipart uploads under a prefix.

Specified by:

newListMultipartUploadsRequest in interface RequestFactory

Parameters:

prefix - prefix to list under

Returns:

the request.

newAbortMultipartUploadRequest

public com.amazonaws.services.s3.model.AbortMultipartUploadRequest newAbortMultipartUploadRequest(String destKey,
                                                                                                  String uploadId)

Description copied from interface: RequestFactory

Abort a multipart upload.

Specified by:

newAbortMultipartUploadRequest in interface RequestFactory

Parameters:

destKey - destination object key

uploadId - ID of initiated upload

Returns:

the request.

newMultipartUploadRequest

public com.amazonaws.services.s3.model.InitiateMultipartUploadRequest newMultipartUploadRequest(String destKey)

Description copied from interface: RequestFactory

Start a multipart upload.

Specified by:

newMultipartUploadRequest in interface RequestFactory

Parameters:

destKey - destination object key

Returns:

the request.

newCompleteMultipartUploadRequest

public com.amazonaws.services.s3.model.CompleteMultipartUploadRequest newCompleteMultipartUploadRequest(String destKey,
                                                                                                        String uploadId,
                                                                                                        List<com.amazonaws.services.s3.model.PartETag> partETags)

Description copied from interface: RequestFactory

Complete a multipart upload.

Specified by:

newCompleteMultipartUploadRequest in interface RequestFactory

Parameters:

destKey - destination object key

uploadId - ID of initiated upload

partETags - ordered list of etags

Returns:

the request.

newGetObjectMetadataRequest

public com.amazonaws.services.s3.model.GetObjectMetadataRequest newGetObjectMetadataRequest(String key)

Description copied from interface: RequestFactory

Create a HEAD request.

Specified by:

newGetObjectMetadataRequest in interface RequestFactory

Parameters:

key - key, may have trailing /

Returns:

the request.

newGetObjectRequest

public com.amazonaws.services.s3.model.GetObjectRequest newGetObjectRequest(String key)

Description copied from interface: RequestFactory

Create a GET request.

Specified by:

newGetObjectRequest in interface RequestFactory

Parameters:

key - object key

Returns:

the request.

newUploadPartRequest

public com.amazonaws.services.s3.model.UploadPartRequest newUploadPartRequest(String destKey,
                                                                              String uploadId,
                                                                              int partNumber,
                                                                              int size,
                                                                              InputStream uploadStream,
                                                                              File sourceFile,
                                                                              long offset)
                                                                       throws org.apache.hadoop.fs.PathIOException

Description copied from interface: RequestFactory

Create and initialize a part request of a multipart upload. Exactly one of: uploadStream or sourceFile must be specified. A subset of the file may be posted, by providing the starting point in offset and a length of block in size equal to or less than the remaining bytes.

Specified by:

newUploadPartRequest in interface RequestFactory

Parameters:

destKey - destination key of ongoing operation

uploadId - ID of ongoing upload

partNumber - current part number of the upload

size - amount of data

uploadStream - source of data to upload

sourceFile - optional source file.

offset - offset in file to start reading.

Returns:

the request.

Throws:

org.apache.hadoop.fs.PathIOException - if the part number is out of range.

newSelectRequest

public com.amazonaws.services.s3.model.SelectObjectContentRequest newSelectRequest(String key)

Description copied from interface: RequestFactory

Create a S3 Select request for the destination object. This does not build the query.

Specified by:

newSelectRequest in interface RequestFactory

Parameters:

key - object key

Returns:

the request

newListObjectsV1Request

public com.amazonaws.services.s3.model.ListObjectsRequest newListObjectsV1Request(String key,
                                                                                  String delimiter,
                                                                                  int maxKeys)

Description copied from interface: RequestFactory

Create the (legacy) V1 list request.

Specified by:

newListObjectsV1Request in interface RequestFactory

Parameters:

key - key to list under

delimiter - delimiter for keys

maxKeys - maximum number in a list page.

Returns:

the request

newListNextBatchOfObjectsRequest

public com.amazonaws.services.s3.model.ListNextBatchOfObjectsRequest newListNextBatchOfObjectsRequest(com.amazonaws.services.s3.model.ObjectListing prev)

Description copied from interface: RequestFactory

Create the next V1 page list request, following on from the previous response.

Specified by:

newListNextBatchOfObjectsRequest in interface RequestFactory

Parameters:

prev - previous response

Returns:

the request

newListObjectsV2Request

public com.amazonaws.services.s3.model.ListObjectsV2Request newListObjectsV2Request(String key,
                                                                                    String delimiter,
                                                                                    int maxKeys)

Description copied from interface: RequestFactory

Create a V2 list request. This will be recycled for any subsequent requests.

Specified by:

newListObjectsV2Request in interface RequestFactory

Parameters:

key - key to list under

delimiter - delimiter for keys

maxKeys - maximum number in a list page.

Returns:

the request

newDeleteObjectRequest

public com.amazonaws.services.s3.model.DeleteObjectRequest newDeleteObjectRequest(String key)

Description copied from interface: RequestFactory

Create a request to delete a single object.

Specified by:

newDeleteObjectRequest in interface RequestFactory

Parameters:

key - object to delete

Returns:

the request

newBulkDeleteRequest

public com.amazonaws.services.s3.model.DeleteObjectsRequest newBulkDeleteRequest(List<com.amazonaws.services.s3.model.DeleteObjectsRequest.KeyVersion> keysToDelete,
                                                                                 boolean quiet)

Description copied from interface: RequestFactory

Bulk delete request.

Specified by:

newBulkDeleteRequest in interface RequestFactory

Parameters:

keysToDelete - list of keys to delete.

quiet - should a bulk query be quiet, or should its result list all deleted keys?

Returns:

the request

setEncryptionSecrets

public void setEncryptionSecrets(EncryptionSecrets secrets)

Description copied from interface: RequestFactory

Set the encryption secrets for all subsequent requests.

Specified by:

setEncryptionSecrets in interface RequestFactory

Parameters:

secrets - encryption secrets.

builder

public static RequestFactoryImpl.RequestFactoryBuilder builder()

Create a builder.

Returns:

new builder.