Package software.amazon.awssdk.transfer.s3

Interface S3TransferManager

All Superinterfaces:

AutoCloseable, SdkAutoCloseable

@ThreadSafe
public interface S3TransferManager
extends SdkAutoCloseable

The S3 Transfer Manager offers a simple API that allows you to transfer a single object or a set of objects to and from Amazon S3 with enhanced throughput and reliability. It leverages Amazon S3 multipart upload and byte-range fetches to perform transfers in parallel. In addition, the S3 Transfer Manager also enables you to monitor a transfer's progress in real-time, as well as pause the transfer for execution at a later time.

Instantiate the S3 Transfer Manager

Create a transfer manager instance with SDK default settings. Note that it's recommended to add software.amazon.awssdk.crt:aws-crt dependency so that automatic multipart feature can be enabled Create an S3 Transfer Manager instance with custom settings Create an S3 Transfer Manager with S3 Multipart Async Client. The S3 Multipart Async Client is an alternative to the CRT client that offers the same multipart upload/download feature.

Common Usage Patterns

Upload a file to S3 Download an S3 object to a local file Upload a local directory to an S3 bucket Download S3 objects to a local directory Copy an S3 object to a different location in S3 The automatic parallel transfer feature (multipart upload/download) is available through the AWS-CRT based S3 client S3AsyncClient.crtBuilder().build) and Java-based S3 multipart client S3AsyncClient.builder().multipartEnabled(true).build(). If no client is configured, AWS-CRT based S3 client will be used if AWS CRT is in the classpath, otherwise, Java-based S3 multipart client will be used.

Nested Class Summary

Nested Classes

Modifier and Type	Interface	Description
static interface	S3TransferManager.Builder	The builder definition for a S3TransferManager.

Method Summary

Modifier and Type	Method	Description
static S3TransferManager.Builder	builder()	Creates a default builder for S3TransferManager.
default Copy	copy(Consumer<CopyRequest.Builder> copyRequestBuilder)	This is a convenience method that creates an instance of the CopyRequest builder, avoiding the need to create one manually via CopyRequest.builder().
default Copy	copy(CopyRequest copyRequest)	Creates a copy of an object that is already stored in S3.
static S3TransferManager	create()	Create an S3TransferManager using the default values.
default <ResultT> Download<ResultT>	download(DownloadRequest<ResultT> downloadRequest)	Downloads an object identified by the bucket and key from S3 through the given AsyncResponseTransformer.
default DirectoryDownload	downloadDirectory(Consumer<DownloadDirectoryRequest.Builder> requestBuilder)	This is a convenience method that creates an instance of the DownloadDirectoryRequest builder, avoiding the need to create one manually via DownloadDirectoryRequest.builder().
default DirectoryDownload	downloadDirectory(DownloadDirectoryRequest downloadDirectoryRequest)	Downloads all objects under a bucket to the provided directory.
default FileDownload	downloadFile(Consumer<DownloadFileRequest.Builder> request)	This is a convenience method that creates an instance of the DownloadFileRequest builder, avoiding the need to create one manually via DownloadFileRequest.builder().
default FileDownload	downloadFile(DownloadFileRequest downloadRequest)	Downloads an object identified by the bucket and key from S3 to a local file.
default FileDownload	resumeDownloadFile(Consumer<ResumableFileDownload.Builder> resumableFileDownload)	This is a convenience method that creates an instance of the ResumableFileDownload builder, avoiding the need to create one manually via ResumableFileDownload.builder().
default FileDownload	resumeDownloadFile(ResumableFileDownload resumableFileDownload)	Resumes a downloadFile operation.
default FileUpload	resumeUploadFile(Consumer<ResumableFileUpload.Builder> resumableFileUpload)	This is a convenience method that creates an instance of the ResumableFileUpload builder, avoiding the need to create one manually via ResumableFileUpload.builder().
default FileUpload	resumeUploadFile(ResumableFileUpload resumableFileUpload)	Resumes uploadFile operation.
default Upload	upload(Consumer<UploadRequest.Builder> request)	This is a convenience method that creates an instance of the UploadRequest builder, avoiding the need to create one manually via UploadRequest.builder().
default Upload	upload(UploadRequest uploadRequest)	Uploads the given AsyncRequestBody to an object in S3.
default DirectoryUpload	uploadDirectory(Consumer<UploadDirectoryRequest.Builder> requestBuilder)	This is a convenience method that creates an instance of the UploadDirectoryRequest builder, avoiding the need to create one manually via UploadDirectoryRequest.builder().
default DirectoryUpload	uploadDirectory(UploadDirectoryRequest uploadDirectoryRequest)	Uploads all files under the given directory to the provided S3 bucket.
default FileUpload	uploadFile(Consumer<UploadFileRequest.Builder> request)	This is a convenience method that creates an instance of the UploadFileRequest builder, avoiding the need to create one manually via UploadFileRequest.builder().
default FileUpload	uploadFile(UploadFileRequest uploadFileRequest)	Uploads a local file to an object in S3.

Methods inherited from interface software.amazon.awssdk.utils.SdkAutoCloseable

close

Method Detail

downloadFile

default FileDownload downloadFile(DownloadFileRequest downloadRequest)

Downloads an object identified by the bucket and key from S3 to a local file. For non-file-based downloads, you may use download(DownloadRequest) instead.

The SDK will create a new file if the provided one doesn't exist. The default permission for the new file depends on the file system and platform. Users can configure the permission on the file using Java API by themselves. If the file already exists, the SDK will replace it. In the event of an error, the SDK will NOT attempt to delete the file, leaving it as-is.

Users can monitor the progress of the transfer by attaching a TransferListener. The provided LoggingTransferListener logs a basic progress bar; users can also implement their own listeners.

Usage Example:

See Also:

downloadFile(Consumer), download(DownloadRequest)

downloadFile

default FileDownload downloadFile(Consumer<DownloadFileRequest.Builder> request)

This is a convenience method that creates an instance of the DownloadFileRequest builder, avoiding the need to create one manually via DownloadFileRequest.builder().

See Also:

downloadFile(DownloadFileRequest)

resumeDownloadFile

default FileDownload resumeDownloadFile(ResumableFileDownload resumableFileDownload)

Resumes a downloadFile operation. This download operation uses the same configuration as the original download. Any content that has already been fetched since the last pause will be skipped and only the remaining data will be downloaded from Amazon S3.

If it is determined that the source S3 object or the destination file has be modified since the last pause, the SDK will download the object from the beginning as if it is a new DownloadFileRequest.

Usage Example:

Parameters:

resumableFileDownload - the download to resume.

Returns:

A new FileDownload object to use to check the state of the download.

See Also:

downloadFile(DownloadFileRequest), FileDownload.pause()

resumeDownloadFile

default FileDownload resumeDownloadFile(Consumer<ResumableFileDownload.Builder> resumableFileDownload)

This is a convenience method that creates an instance of the ResumableFileDownload builder, avoiding the need to create one manually via ResumableFileDownload.builder().

See Also:

resumeDownloadFile(ResumableFileDownload)

download

default <ResultT> Download<ResultT> download(DownloadRequest<ResultT> downloadRequest)

Downloads an object identified by the bucket and key from S3 through the given AsyncResponseTransformer. For downloading to a file, you may use downloadFile(DownloadFileRequest) instead.

Users can monitor the progress of the transfer by attaching a TransferListener. The provided LoggingTransferListener logs a basic progress bar; users can also implement their own listeners.

Usage Example (this example buffers the entire object in memory and is not suitable for large objects):

See the static factory methods available in AsyncResponseTransformer for other use cases.

Type Parameters:

ResultT - The type of data the AsyncResponseTransformer produces

Parameters:

downloadRequest - the download request, containing a GetObjectRequest and AsyncResponseTransformer

Returns:

A Download that can be used to track the ongoing transfer

See Also:

downloadFile(DownloadFileRequest)

uploadFile

default FileUpload uploadFile(UploadFileRequest uploadFileRequest)

Uploads a local file to an object in S3. For non-file-based uploads, you may use upload(UploadRequest) instead.

Users can monitor the progress of the transfer by attaching a TransferListener. The provided LoggingTransferListener logs a basic progress bar; users can also implement their own listeners. Upload a local file to an object in S3. For non-file-based uploads, you may use upload(UploadRequest) instead.

Usage Example:

See Also:

uploadFile(Consumer), upload(UploadRequest)

uploadFile

default FileUpload uploadFile(Consumer<UploadFileRequest.Builder> request)

This is a convenience method that creates an instance of the UploadFileRequest builder, avoiding the need to create one manually via UploadFileRequest.builder().

See Also:

uploadFile(UploadFileRequest)

resumeUploadFile

default FileUpload resumeUploadFile(ResumableFileUpload resumableFileUpload)

Resumes uploadFile operation. This upload operation will use the same configuration provided in ResumableFileUpload. The SDK will skip the data that has already been upload since the last pause and only upload the remaining data from the source file.

If it is determined that the source file has be modified since the last pause, the SDK will upload the object from the beginning as if it is a new UploadFileRequest.

Usage Example:

Parameters:

resumableFileUpload - the upload to resume.

Returns:

A new FileUpload object to use to check the state of the download.

See Also:

uploadFile(UploadFileRequest), FileUpload.pause()

resumeUploadFile

default FileUpload resumeUploadFile(Consumer<ResumableFileUpload.Builder> resumableFileUpload)

This is a convenience method that creates an instance of the ResumableFileUpload builder, avoiding the need to create one manually via ResumableFileUpload.builder().

See Also:

resumeUploadFile(ResumableFileUpload)

upload

default Upload upload(UploadRequest uploadRequest)

Uploads the given AsyncRequestBody to an object in S3. For file-based uploads, you may use uploadFile(UploadFileRequest) instead.

Users can monitor the progress of the transfer by attaching a TransferListener. The provided LoggingTransferListener logs a basic progress bar; users can also implement their own listeners.

Usage Example: See the static factory methods available in AsyncRequestBody for other use cases.

Parameters:

uploadRequest - the upload request, containing a PutObjectRequest and AsyncRequestBody

Returns:

An Upload that can be used to track the ongoing transfer

See Also:

upload(Consumer), uploadFile(UploadFileRequest)

upload

default Upload upload(Consumer<UploadRequest.Builder> request)

This is a convenience method that creates an instance of the UploadRequest builder, avoiding the need to create one manually via UploadRequest.builder().

See Also:

upload(UploadRequest)

uploadDirectory

default DirectoryUpload uploadDirectory(UploadDirectoryRequest uploadDirectoryRequest)

Uploads all files under the given directory to the provided S3 bucket. The key name transformation depends on the optional prefix and delimiter provided in the UploadDirectoryRequest. By default, all subdirectories will be uploaded recursively, and symbolic links are not followed automatically. This behavior can be configured in at request level via UploadDirectoryRequest.Builder.followSymbolicLinks(Boolean) or client level via S3TransferManager.Builder.uploadDirectoryFollowSymbolicLinks(Boolean) Note that request-level configuration takes precedence over client-level configuration.

By default, the prefix is an empty string and the delimiter is "/". Assume you have a local directory "/test" with the following structure:

   
      |- test
         |- sample.jpg
         |- photos
             |- 2022
                 |- January
                     |- sample.jpg
                 |- February
                     |- sample1.jpg
                     |- sample2.jpg
                     |- sample3.jpg
   
 

Give a request to upload directory "/test" to an S3 bucket, the target bucket will have the following S3 objects:

• sample.jpg
• photos/2022/January/sample.jpg
• photos/2022/February/sample1.jpg
• photos/2022/February/sample2.jpg
• photos/2022/February/sample3.jpg

The returned CompletableFuture only completes exceptionally if the request cannot be attempted as a whole (the source directory provided does not exist for example). The future completes successfully for partial successful requests, i.e., there might be failed uploads in the successfully completed response. As a result, you should check for errors in the response via CompletedDirectoryUpload.failedTransfers() even when the future completes successfully.

The current user must have read access to all directories and files.

Usage Example:

Parameters:

uploadDirectoryRequest - the upload directory request

See Also:

uploadDirectory(Consumer)

uploadDirectory

default DirectoryUpload uploadDirectory(Consumer<UploadDirectoryRequest.Builder> requestBuilder)

This is a convenience method that creates an instance of the UploadDirectoryRequest builder, avoiding the need to create one manually via UploadDirectoryRequest.builder().

See Also:

uploadDirectory(UploadDirectoryRequest)

downloadDirectory

default DirectoryDownload downloadDirectory(DownloadDirectoryRequest downloadDirectoryRequest)

Downloads all objects under a bucket to the provided directory. By default, all objects in the entire bucket will be downloaded. You can modify this behavior by providing a DownloadDirectoryRequest.listObjectsRequestTransformer() and/or a DownloadDirectoryRequest.filter() in DownloadDirectoryRequest to limit the S3 objects to download.

The downloaded directory structure will match with the provided S3 virtual bucket. For example, assume that you have the following keys in your bucket:

• sample.jpg
• photos/2022/January/sample.jpg
• photos/2022/February/sample1.jpg
• photos/2022/February/sample2.jpg
• photos/2022/February/sample3.jpg

Give a request to download the bucket to a destination with path of "/test", the downloaded directory would look like this

   
      |- test
         |- sample.jpg
         |- photos
             |- 2022
                 |- January
                     |- sample.jpg
                 |- February
                     |- sample1.jpg
                     |- sample2.jpg
                     |- sample3.jpg
   
 

The returned CompletableFuture only completes exceptionally if the request cannot be attempted as a whole (the downloadDirectoryRequest is invalid for example). The future completes successfully for partial successful requests, i.e., there might be failed downloads in a successfully completed response. As a result, you should check for errors in the response via CompletedDirectoryDownload.failedTransfers() even when the future completes successfully.

The SDK will create the destination directory if it does not already exist. If a specific file already exists, the existing content will be replaced with the corresponding S3 object content.

The current user must have write access to all directories and files

Usage Example:

Parameters:

downloadDirectoryRequest - the download directory request

See Also:

downloadDirectory(Consumer)

downloadDirectory

default DirectoryDownload downloadDirectory(Consumer<DownloadDirectoryRequest.Builder> requestBuilder)

This is a convenience method that creates an instance of the DownloadDirectoryRequest builder, avoiding the need to create one manually via DownloadDirectoryRequest.builder().

See Also:

downloadDirectory(DownloadDirectoryRequest)

copy

default Copy copy(CopyRequest copyRequest)

Creates a copy of an object that is already stored in S3.

Depending on the underlying S3Client, S3TransferManager may intelligently use plain CopyObjectRequests for smaller objects, and multiple parallel UploadPartCopyRequests for larger objects. If multipart copy is supported by the underlying S3Client, this behavior can be configured via S3CrtAsyncClientBuilder.minimumPartSizeInBytes(Long). Note that for multipart copy request, existing metadata stored in the source object is NOT copied to the destination object; if required, you can retrieve the metadata from the source object and set it explicitly in the @link CopyObjectRequest.Builder#metadata(Map)}.

While this API supports TransferListeners, they will not receive bytesTransferred callback-updates due to the way the CopyObjectRequest API behaves. When copying an object, S3 performs the byte copying on your behalf while keeping the connection alive. The progress of the copy is not known until it fully completes and S3 sends a response describing the outcome.

If you are copying an object to a bucket in a different region, you need to enable cross region access on the S3AsyncClient.

Usage Example:

Parameters:

copyRequest - the copy request, containing a CopyObjectRequest

Returns:

A Copy that can be used to track the ongoing transfer

See Also:

copy(Consumer), S3AsyncClient.copyObject(CopyObjectRequest)

copy

default Copy copy(Consumer<CopyRequest.Builder> copyRequestBuilder)

This is a convenience method that creates an instance of the CopyRequest builder, avoiding the need to create one manually via CopyRequest.builder().

See Also:

copy(CopyRequest)

create

static S3TransferManager create()

Create an S3TransferManager using the default values.

The type of S3AsyncClient used depends on if AWS Common Runtime (CRT) library software.amazon.awssdk.crt:aws-crt is in the classpath. If AWS CRT is available, an AWS CRT-based S3 client will be created via (S3AsyncClient.crtCreate()). Otherwise, a standard S3 client(S3AsyncClient.create()) will be created. Note that only AWS CRT-based S3 client supports parallel transfer, i.e, leveraging multipart upload/download for now, so it's recommended to add AWS CRT as a dependency.

builder

static S3TransferManager.Builder builder()

Creates a default builder for S3TransferManager.