public class S3OutputStream
Facilitates streaming to S3 in case the actual file size is not known upfront: uploading via the regular single-file upload will cache the contents in memory and cause an out-of-memory error, especially on AWS lambda. This may happen if you have to stream several GB to S3.
This class provides an OutputStream to write to. It will initiate a multipart upload and thereby limit the amount of data currently in memory.
Upon closing this stream, the upload to S3 will be completed. Note
that, for this reason, the S3OutputStream.close method may block and throw all
sorts of exceptions.
This class is NOT thread safe!
Internally, there are two buffers. One is used to buffer incoming writes. As soon as that is full, it will be switched over for an empty one. Then, the contents of the filled buffer are uploaded in parallel. If the write buffer is filled before the other buffer has been uploaded, the stream blocks until the upload is completed.
This means that, for optimal performance, you should set the
S3OutputStream.close| Modifier and Type | Class and Description |
|---|---|
static class |
S3OutputStream.Companion |
| Modifier and Type | Field and Description |
|---|---|
static S3OutputStream.Companion |
Companion |
static long |
MAX_SINGLE_FILE_UPLOAD_SIZE
The maximum size for a single file upload. This value is defined by
the AWS S3 people.
|
static long |
MAX_UPLOAD_PART_SIZE
The maximum size of a single part in a multipart upload. This
value is defined by the AWS S3 people.
|
static int |
MIN_UPLOAD_PART_SIZE
The minimum size for single parts of multipart uploads. This
value is defined by the AWS S3 people.
|
| Constructor and Description |
|---|
S3OutputStream(com.amazonaws.services.s3.AmazonS3 awsS3,
java.lang.String targetBucket,
java.lang.String targetS3Key,
long maxLocalCache,
boolean useChecksums,
ByteBufferPool byteBufferPool)
Facilitates streaming to S3 in case the actual file size is not
known upfront: uploading via the regular single-file upload will
cache the contents in memory and cause an out-of-memory error,
especially on AWS lambda. This may happen if you have to stream
several GB to S3.
|
| Modifier and Type | Method and Description |
|---|---|
void |
close()
Fully flushes the stream and completes the upload.
|
void |
flush()
Flushes this ONLY if there is enough data in the buffer,
see MIN_UPLOAD_PART_SIZE
|
long |
getActualLocalCache()
The actual amount of local cache; this caps the requested maxLocalCache to what
the ByteBuffers can hold (2GiB each).
|
com.amazonaws.services.s3.AmazonS3 |
getAwsS3()
The client to upload with
|
long |
getMaxLocalCache()
The maximum amount of memory to use as buffer,
|
java.lang.String |
getTargetBucket() |
java.lang.String |
getTargetS3Key() |
boolean |
getUseChecksums()
Whether to calculate MD5 checksums of the uploaded data.
|
void |
write(int b) |
void |
write(byte[] b,
int off,
int len) |
public static int MIN_UPLOAD_PART_SIZE
The minimum size for single parts of multipart uploads. This value is defined by the AWS S3 people.
See https//docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html
public static long MAX_UPLOAD_PART_SIZE
The maximum size of a single part in a multipart upload. This value is defined by the AWS S3 people.
See https//docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html
public static long MAX_SINGLE_FILE_UPLOAD_SIZE
The maximum size for a single file upload. This value is defined by the AWS S3 people.
See https//docs.aws.amazon.com/AmazonS3/latest/dev/UploadingObjects.html
public static S3OutputStream.Companion Companion
public S3OutputStream(com.amazonaws.services.s3.AmazonS3 awsS3,
java.lang.String targetBucket,
java.lang.String targetS3Key,
long maxLocalCache,
boolean useChecksums,
ByteBufferPool byteBufferPool)
Facilitates streaming to S3 in case the actual file size is not known upfront: uploading via the regular single-file upload will cache the contents in memory and cause an out-of-memory error, especially on AWS lambda. This may happen if you have to stream several GB to S3.
This class provides an OutputStream to write to. It will initiate a multipart upload and thereby limit the amount of data currently in memory.
Upon closing this stream, the upload to S3 will be completed. Note
that, for this reason, the S3OutputStream.close method may block and throw all
sorts of exceptions.
This class is NOT thread safe!
Internally, there are two buffers. One is used to buffer incoming writes. As soon as that is full, it will be switched over for an empty one. Then, the contents of the filled buffer are uploaded in parallel. If the write buffer is filled before the other buffer has been uploaded, the stream blocks until the upload is completed.
This means that, for optimal performance, you should set the
awsS3 - The client to upload withmaxLocalCache - The maximum amount of memory to use as buffer, in bytesuseChecksums - Whether to calculate MD5 checksums of the uploaded data. Turn thisoff if you are concerned about the CPU load.byteBufferPool - The buffers for local cache are obtained from here. Pass a custom implementationif you want to change the behaviour or share memory.S3OutputStream.closepublic long getActualLocalCache()
The actual amount of local cache; this caps the requested maxLocalCache to what the ByteBuffers can hold (2GiB each).
public void write(int b)
public void write(byte[] b,
int off,
int len)
public void flush()
Flushes this ONLY if there is enough data in the buffer, see MIN_UPLOAD_PART_SIZE
public void close()
Fully flushes the stream and completes the upload.
public com.amazonaws.services.s3.AmazonS3 getAwsS3()
The client to upload with
public java.lang.String getTargetBucket()
public java.lang.String getTargetS3Key()
public long getMaxLocalCache()
The maximum amount of memory to use as buffer,
in bytes
public boolean getUseChecksums()
Whether to calculate MD5 checksums of the uploaded data.
Turn thisoff if you are concerned about the CPU load.