com.hubject.aws.s3.io

Class S3OutputStream

com.hubject.aws.s3.io.S3OutputStream

public class S3OutputStream

Facilitates streaming to S3 in case the actual file size is not known upfront: uploading via the regular single-file upload will cache the contents in memory and cause an out-of-memory error, especially on AWS lambda. This may happen if you have to stream several GB to S3.

This class provides an OutputStream to write to. It will initiate a multipart upload and thereby limit the amount of data currently in memory.

Upon closing this stream, the upload to S3 will be completed. Note that, for this reason, the S3OutputStream.close method may block and throw all sorts of exceptions.

This class is NOT thread safe!

Internally, there are two buffers. One is used to buffer incoming writes. As soon as that is full, it will be switched over for an empty one. Then, the contents of the filled buffer are uploaded in parallel. If the write buffer is filled before the other buffer has been uploaded, the stream blocks until the upload is completed.

This means that, for optimal performance, you should set the

See Also:

S3OutputStream.close

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	S3OutputStream.Companion

Field Summary

Fields

Modifier and Type	Field and Description
static S3OutputStream.Companion	Companion
static long	MAX_SINGLE_FILE_UPLOAD_SIZE The maximum size for a single file upload. This value is defined by the AWS S3 people.
static long	MAX_UPLOAD_PART_SIZE The maximum size of a single part in a multipart upload. This value is defined by the AWS S3 people.
static int	MIN_UPLOAD_PART_SIZE The minimum size for single parts of multipart uploads. This value is defined by the AWS S3 people.

Constructor Summary

Constructors

Constructor and Description

S3OutputStream(com.amazonaws.services.s3.AmazonS3 awsS3, java.lang.String targetBucket, java.lang.String targetS3Key, long maxLocalCache, boolean useChecksums, ByteBufferPool byteBufferPool) Facilitates streaming to S3 in case the actual file size is not known upfront: uploading via the regular single-file upload will cache the contents in memory and cause an out-of-memory error, especially on AWS lambda. This may happen if you have to stream several GB to S3.

Method Summary

Modifier and Type	Method and Description
void	close() Fully flushes the stream and completes the upload.
void	flush() Flushes this ONLY if there is enough data in the buffer, see MIN_UPLOAD_PART_SIZE
long	getActualLocalCache() The actual amount of local cache; this caps the requested maxLocalCache to what the ByteBuffers can hold (2GiB each).
com.amazonaws.services.s3.AmazonS3	getAwsS3() The client to upload with
long	getMaxLocalCache() The maximum amount of memory to use as buffer,
java.lang.String	getTargetBucket()
java.lang.String	getTargetS3Key()
boolean	getUseChecksums() Whether to calculate MD5 checksums of the uploaded data.
void	write(int b)
void	write(byte[] b, int off, int len)

Field Detail

MIN_UPLOAD_PART_SIZE

public static int MIN_UPLOAD_PART_SIZE

The minimum size for single parts of multipart uploads. This value is defined by the AWS S3 people.

See https//docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html

See Also:

https//docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html

MAX_UPLOAD_PART_SIZE

public static long MAX_UPLOAD_PART_SIZE

The maximum size of a single part in a multipart upload. This value is defined by the AWS S3 people.

See https//docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html

See Also:

https//docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html

MAX_SINGLE_FILE_UPLOAD_SIZE

public static long MAX_SINGLE_FILE_UPLOAD_SIZE

The maximum size for a single file upload. This value is defined by the AWS S3 people.

See https//docs.aws.amazon.com/AmazonS3/latest/dev/UploadingObjects.html

See Also:

https//docs.aws.amazon.com/AmazonS3/latest/dev/UploadingObjects.html

Companion

public static S3OutputStream.Companion Companion

Constructor Detail

S3OutputStream

public S3OutputStream(com.amazonaws.services.s3.AmazonS3 awsS3,
                      java.lang.String targetBucket,
                      java.lang.String targetS3Key,
                      long maxLocalCache,
                      boolean useChecksums,
                      ByteBufferPool byteBufferPool)

Facilitates streaming to S3 in case the actual file size is not known upfront: uploading via the regular single-file upload will cache the contents in memory and cause an out-of-memory error, especially on AWS lambda. This may happen if you have to stream several GB to S3.

This class provides an OutputStream to write to. It will initiate a multipart upload and thereby limit the amount of data currently in memory.

Upon closing this stream, the upload to S3 will be completed. Note that, for this reason, the S3OutputStream.close method may block and throw all sorts of exceptions.

This class is NOT thread safe!

Internally, there are two buffers. One is used to buffer incoming writes. As soon as that is full, it will be switched over for an empty one. Then, the contents of the filled buffer are uploaded in parallel. If the write buffer is filled before the other buffer has been uploaded, the stream blocks until the upload is completed.

This means that, for optimal performance, you should set the

Parameters:

awsS3 - The client to upload with

maxLocalCache - The maximum amount of memory to use as buffer, in bytes

useChecksums - Whether to calculate MD5 checksums of the uploaded data. Turn thisoff if you are concerned about the CPU load.

byteBufferPool - The buffers for local cache are obtained from here. Pass a custom implementationif you want to change the behaviour or share memory.

See Also:

S3OutputStream.close

Method Detail

getActualLocalCache

public long getActualLocalCache()

The actual amount of local cache; this caps the requested maxLocalCache to what the ByteBuffers can hold (2GiB each).

See Also:

ByteBuffer

write

public void write(int b)

write

public void write(byte[] b,
                  int off,
                  int len)

flush

public void flush()

Flushes this ONLY if there is enough data in the buffer, see MIN_UPLOAD_PART_SIZE

close

public void close()

Fully flushes the stream and completes the upload.

getAwsS3

public com.amazonaws.services.s3.AmazonS3 getAwsS3()

The client to upload with

getTargetBucket

public java.lang.String getTargetBucket()

getTargetS3Key

public java.lang.String getTargetS3Key()

getMaxLocalCache

public long getMaxLocalCache()

The maximum amount of memory to use as buffer,

in bytes

getUseChecksums

public boolean getUseChecksums()

Whether to calculate MD5 checksums of the uploaded data.

Turn thisoff if you are concerned about the CPU load.