org.apache.hadoop.fs.s3a.impl

Class RequestFactoryImpl

java.lang.Object

org.apache.hadoop.fs.s3a.impl.RequestFactoryImpl

All Implemented Interfaces:

RequestFactory

public class RequestFactoryImpl
extends Object
implements RequestFactory

The standard implementation of the request factory. This creates AWS SDK request classes for the specific bucket, with standard options/headers set. It is also where custom setting parameters can take place. All creation of AWS S3 requests MUST be through this class so that common options (encryption etc.) can be added here, and so that any chained transformation of requests can be applied. This is where audit span information is added to the requests, until it is done in the AWS SDK itself. All created request builders will be passed to RequestFactoryImpl.PrepareRequest.prepareRequest(SdkRequest.Builder) before being returned to the caller.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	RequestFactoryImpl.PrepareRequest This is a callback for anything to "prepare" every request after creation.
static class	RequestFactoryImpl.RequestFactoryBuilder Builder.

Field Summary

Fields

Modifier and Type	Field and Description
static org.slf4j.Logger	LOG

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	RequestFactoryImpl(RequestFactoryImpl.RequestFactoryBuilder builder) Constructor.

Method Summary

Modifier and Type	Method and Description
static RequestFactoryImpl.RequestFactoryBuilder	builder() Create a builder.
protected void	copyEncryptionParameters(software.amazon.awssdk.services.s3.model.HeadObjectResponse srcom, software.amazon.awssdk.services.s3.model.CopyObjectRequest.Builder copyObjectRequestBuilder) Propagate encryption parameters from source file if set else use the current filesystem encryption settings.
protected String	getBucket() Get the target bucket.
String	getCannedACL() Get the canned ACL of this FS.
String	getContentEncoding() Get the content encoding (e.g.
S3AEncryptionMethods	getServerSideEncryptionAlgorithm() Get the encryption algorithm of this endpoint.
software.amazon.awssdk.services.s3.model.StorageClass	getStorageClass() Get the object storage class, return null if none.
software.amazon.awssdk.services.s3.model.AbortMultipartUploadRequest.Builder	newAbortMultipartUploadRequestBuilder(String destKey, String uploadId) Abort a multipart upload.
software.amazon.awssdk.services.s3.model.DeleteObjectsRequest.Builder	newBulkDeleteRequestBuilder(List<software.amazon.awssdk.services.s3.model.ObjectIdentifier> keysToDelete) Create a request builder to delete objects in bulk.
software.amazon.awssdk.services.s3.model.CompleteMultipartUploadRequest.Builder	newCompleteMultipartUploadRequestBuilder(String destKey, String uploadId, List<software.amazon.awssdk.services.s3.model.CompletedPart> partETags) Complete a multipart upload.
software.amazon.awssdk.services.s3.model.CopyObjectRequest.Builder	newCopyObjectRequestBuilder(String srcKey, String dstKey, software.amazon.awssdk.services.s3.model.HeadObjectResponse srcom) Create a copy request builder.
software.amazon.awssdk.services.s3.model.DeleteObjectRequest.Builder	newDeleteObjectRequestBuilder(String key) Create a request builder to delete a single object.
software.amazon.awssdk.services.s3.model.PutObjectRequest.Builder	newDirectoryMarkerRequest(String directory) Create a PutObjectRequest request for creating an empty directory.
software.amazon.awssdk.services.s3.model.GetObjectRequest.Builder	newGetObjectRequestBuilder(String key) Create a GET request builder.
software.amazon.awssdk.services.s3.model.HeadBucketRequest.Builder	newHeadBucketRequestBuilder(String bucketName) Create a HEAD bucket request builder.
software.amazon.awssdk.services.s3.model.HeadObjectRequest.Builder	newHeadObjectRequestBuilder(String key) Create a HEAD object request builder.
software.amazon.awssdk.services.s3.model.ListMultipartUploadsRequest.Builder	newListMultipartUploadsRequestBuilder(String prefix) List all multipart uploads under a prefix.
software.amazon.awssdk.services.s3.model.ListObjectsRequest.Builder	newListObjectsV1RequestBuilder(String key, String delimiter, int maxKeys) Create the (legacy) V1 list request builder.
software.amazon.awssdk.services.s3.model.ListObjectsV2Request.Builder	newListObjectsV2RequestBuilder(String key, String delimiter, int maxKeys) Create a V2 list request builder.
software.amazon.awssdk.services.s3.model.CreateMultipartUploadRequest.Builder	newMultipartUploadRequestBuilder(String destKey, PutObjectOptions options) Start a multipart upload.
software.amazon.awssdk.services.s3.model.PutObjectRequest.Builder	newPutObjectRequestBuilder(String key, PutObjectOptions options, long length, boolean isDirectoryMarker) Create a putObject request.
software.amazon.awssdk.services.s3.model.UploadPartRequest.Builder	newUploadPartRequestBuilder(String destKey, String uploadId, int partNumber, long size) Create and initialize a part request builder of a multipart upload.
void	setEncryptionSecrets(EncryptionSecrets secrets) Set the encryption secrets for all subsequent requests.
protected void	uploadPartEncryptionParameters(software.amazon.awssdk.services.s3.model.UploadPartRequest.Builder builder) Sets server side encryption parameters to the part upload request when encryption is enabled.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

Constructor Detail

RequestFactoryImpl

protected RequestFactoryImpl(RequestFactoryImpl.RequestFactoryBuilder builder)

Constructor.

Parameters:

builder - builder with all the configuration.

Method Detail

getCannedACL

public String getCannedACL()

Get the canned ACL of this FS.

Specified by:

getCannedACL in interface RequestFactory

Returns:

an ACL, if any

getBucket

protected String getBucket()

Get the target bucket.

Returns:

the bucket.

getServerSideEncryptionAlgorithm

public S3AEncryptionMethods getServerSideEncryptionAlgorithm()

Get the encryption algorithm of this endpoint.

Specified by:

getServerSideEncryptionAlgorithm in interface RequestFactory

Returns:

the encryption algorithm.

getContentEncoding

public String getContentEncoding()

Get the content encoding (e.g. gzip) or return null if none.

Specified by:

getContentEncoding in interface RequestFactory

Returns:

content encoding

getStorageClass

public software.amazon.awssdk.services.s3.model.StorageClass getStorageClass()

Get the object storage class, return null if none.

Specified by:

getStorageClass in interface RequestFactory

Returns:

storage class

uploadPartEncryptionParameters

protected void uploadPartEncryptionParameters(software.amazon.awssdk.services.s3.model.UploadPartRequest.Builder builder)

Sets server side encryption parameters to the part upload request when encryption is enabled.

Parameters:

builder - upload part request builder

newCopyObjectRequestBuilder

public software.amazon.awssdk.services.s3.model.CopyObjectRequest.Builder newCopyObjectRequestBuilder(String srcKey,
                                                                                                      String dstKey,
                                                                                                      software.amazon.awssdk.services.s3.model.HeadObjectResponse srcom)

Description copied from interface: RequestFactory

Create a copy request builder. This includes the work of copying the relevant parts of the metadata from the source

Specified by:

newCopyObjectRequestBuilder in interface RequestFactory

Parameters:

srcKey - source

dstKey - destination

srcom - source object metadata.

Returns:

the request builder

copyEncryptionParameters

protected void copyEncryptionParameters(software.amazon.awssdk.services.s3.model.HeadObjectResponse srcom,
                                        software.amazon.awssdk.services.s3.model.CopyObjectRequest.Builder copyObjectRequestBuilder)

Propagate encryption parameters from source file if set else use the current filesystem encryption settings.

Parameters:

copyObjectRequestBuilder - copy object request builder.

srcom - source object metadata.

newPutObjectRequestBuilder

public software.amazon.awssdk.services.s3.model.PutObjectRequest.Builder newPutObjectRequestBuilder(String key,
                                                                                                    PutObjectOptions options,
                                                                                                    long length,
                                                                                                    boolean isDirectoryMarker)

Create a putObject request. Adds the ACL, storage class and metadata

Specified by:

newPutObjectRequestBuilder in interface RequestFactory

Parameters:

key - key of object

options - options for the request, including headers

length - length of object to be uploaded

isDirectoryMarker - true if object to be uploaded is a directory marker

Returns:

the request builder

newDirectoryMarkerRequest

public software.amazon.awssdk.services.s3.model.PutObjectRequest.Builder newDirectoryMarkerRequest(String directory)

Description copied from interface: RequestFactory

Create a PutObjectRequest request for creating an empty directory.

Specified by:

newDirectoryMarkerRequest in interface RequestFactory

Parameters:

directory - destination directory.

Returns:

request builder for a zero byte upload.

newListMultipartUploadsRequestBuilder

public software.amazon.awssdk.services.s3.model.ListMultipartUploadsRequest.Builder newListMultipartUploadsRequestBuilder(String prefix)

Description copied from interface: RequestFactory

List all multipart uploads under a prefix.

Specified by:

newListMultipartUploadsRequestBuilder in interface RequestFactory

Parameters:

prefix - prefix to list under

Returns:

the request builder.

newAbortMultipartUploadRequestBuilder

public software.amazon.awssdk.services.s3.model.AbortMultipartUploadRequest.Builder newAbortMultipartUploadRequestBuilder(String destKey,
                                                                                                                          String uploadId)

Description copied from interface: RequestFactory

Abort a multipart upload.

Specified by:

newAbortMultipartUploadRequestBuilder in interface RequestFactory

Parameters:

destKey - destination object key

uploadId - ID of initiated upload

Returns:

the request builder.

newMultipartUploadRequestBuilder

public software.amazon.awssdk.services.s3.model.CreateMultipartUploadRequest.Builder newMultipartUploadRequestBuilder(String destKey,
                                                                                                                      @Nullable
                                                                                                                      PutObjectOptions options)
                                                                                                               throws org.apache.hadoop.fs.PathIOException

Description copied from interface: RequestFactory

Start a multipart upload.

Specified by:

newMultipartUploadRequestBuilder in interface RequestFactory

Parameters:

destKey - destination object key

options - options for the request

Returns:

the request builder.

Throws:

org.apache.hadoop.fs.PathIOException - if multipart uploads are disabled

newCompleteMultipartUploadRequestBuilder

public software.amazon.awssdk.services.s3.model.CompleteMultipartUploadRequest.Builder newCompleteMultipartUploadRequestBuilder(String destKey,
                                                                                                                                String uploadId,
                                                                                                                                List<software.amazon.awssdk.services.s3.model.CompletedPart> partETags)

Description copied from interface: RequestFactory

Complete a multipart upload.

Specified by:

newCompleteMultipartUploadRequestBuilder in interface RequestFactory

Parameters:

destKey - destination object key

uploadId - ID of initiated upload

partETags - ordered list of etags

Returns:

the request builder.

newHeadObjectRequestBuilder

public software.amazon.awssdk.services.s3.model.HeadObjectRequest.Builder newHeadObjectRequestBuilder(String key)

Description copied from interface: RequestFactory

Create a HEAD object request builder.

Specified by:

newHeadObjectRequestBuilder in interface RequestFactory

Parameters:

key - key, may have trailing /

Returns:

the request builder.

newHeadBucketRequestBuilder

public software.amazon.awssdk.services.s3.model.HeadBucketRequest.Builder newHeadBucketRequestBuilder(String bucketName)

Description copied from interface: RequestFactory

Create a HEAD bucket request builder.

Specified by:

newHeadBucketRequestBuilder in interface RequestFactory

Parameters:

bucketName - bucket to get metadata for

Returns:

the request builder.

newGetObjectRequestBuilder

public software.amazon.awssdk.services.s3.model.GetObjectRequest.Builder newGetObjectRequestBuilder(String key)

Description copied from interface: RequestFactory

Create a GET request builder.

Specified by:

newGetObjectRequestBuilder in interface RequestFactory

Parameters:

key - object key

Returns:

the request builder.

newUploadPartRequestBuilder

public software.amazon.awssdk.services.s3.model.UploadPartRequest.Builder newUploadPartRequestBuilder(String destKey,
                                                                                                      String uploadId,
                                                                                                      int partNumber,
                                                                                                      long size)
                                                                                               throws org.apache.hadoop.fs.PathIOException

Description copied from interface: RequestFactory

Create and initialize a part request builder of a multipart upload.

Specified by:

newUploadPartRequestBuilder in interface RequestFactory

Parameters:

destKey - destination key of ongoing operation

uploadId - ID of ongoing upload

partNumber - current part number of the upload

size - amount of data

Returns:

the request builder.

Throws:

org.apache.hadoop.fs.PathIOException - if the part number is out of range.

newListObjectsV1RequestBuilder

public software.amazon.awssdk.services.s3.model.ListObjectsRequest.Builder newListObjectsV1RequestBuilder(String key,
                                                                                                          String delimiter,
                                                                                                          int maxKeys)

Description copied from interface: RequestFactory

Create the (legacy) V1 list request builder.

Specified by:

newListObjectsV1RequestBuilder in interface RequestFactory

Parameters:

key - key to list under

delimiter - delimiter for keys

maxKeys - maximum number in a list page.

Returns:

the request builder.

newListObjectsV2RequestBuilder

public software.amazon.awssdk.services.s3.model.ListObjectsV2Request.Builder newListObjectsV2RequestBuilder(String key,
                                                                                                            String delimiter,
                                                                                                            int maxKeys)

Description copied from interface: RequestFactory

Create a V2 list request builder. This will be recycled for any subsequent requests.

Specified by:

newListObjectsV2RequestBuilder in interface RequestFactory

Parameters:

key - key to list under

delimiter - delimiter for keys

maxKeys - maximum number in a list page.

Returns:

the request builder.

newDeleteObjectRequestBuilder

public software.amazon.awssdk.services.s3.model.DeleteObjectRequest.Builder newDeleteObjectRequestBuilder(String key)

Description copied from interface: RequestFactory

Create a request builder to delete a single object.

Specified by:

newDeleteObjectRequestBuilder in interface RequestFactory

Parameters:

key - object to delete

Returns:

the request builder.

newBulkDeleteRequestBuilder

public software.amazon.awssdk.services.s3.model.DeleteObjectsRequest.Builder newBulkDeleteRequestBuilder(List<software.amazon.awssdk.services.s3.model.ObjectIdentifier> keysToDelete)

Description copied from interface: RequestFactory

Create a request builder to delete objects in bulk.

Specified by:

newBulkDeleteRequestBuilder in interface RequestFactory

Parameters:

keysToDelete - list of keys to delete.

Returns:

the request builder.

setEncryptionSecrets

public void setEncryptionSecrets(EncryptionSecrets secrets)

Description copied from interface: RequestFactory

Set the encryption secrets for all subsequent requests.

Specified by:

setEncryptionSecrets in interface RequestFactory

Parameters:

secrets - encryption secrets.

builder

public static RequestFactoryImpl.RequestFactoryBuilder builder()

Create a builder.

Returns:

new builder.