org.apache.hadoop.fs.s3a.impl

Class UploadContentProviders

java.lang.Object

org.apache.hadoop.fs.s3a.impl.UploadContentProviders

public final class UploadContentProviders
extends Object

Implementations of software.amazon.awssdk.http.ContentStreamProvider.

These are required to ensure that retry of multipart uploads are reliable, while also avoiding memory copy/consumption overhead.

For these reasons the providers built in to the AWS SDK are not used.

See HADOOP-19221 for details.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	UploadContentProviders.BaseContentProvider<T extends InputStream> Base class for content providers; tracks the number of times a stream has been opened.

Field Summary

Fields

Modifier and Type	Field and Description
static org.slf4j.Logger	LOG

Method Summary

Modifier and Type	Method and Description
static UploadContentProviders.BaseContentProvider<ByteArrayInputStream>	byteArrayContentProvider(byte[] bytes) Create a content provider for all of a byte array.
static UploadContentProviders.BaseContentProvider<ByteArrayInputStream>	byteArrayContentProvider(byte[] bytes, int offset, int size) Create a content provider for all or part of a byte array.
static UploadContentProviders.BaseContentProvider<ByteArrayInputStream>	byteArrayContentProvider(byte[] bytes, int offset, int size, Supplier<Boolean> isOpen) Create a content provider for all or part of a byte array.
static UploadContentProviders.BaseContentProvider<ByteArrayInputStream>	byteArrayContentProvider(byte[] bytes, Supplier<Boolean> isOpen) Create a content provider for all of a byte array.
static UploadContentProviders.BaseContentProvider<org.apache.hadoop.fs.store.ByteBufferInputStream>	byteBufferContentProvider(ByteBuffer byteBuffer, int size) Create a content provider from a byte buffer.
static UploadContentProviders.BaseContentProvider<org.apache.hadoop.fs.store.ByteBufferInputStream>	byteBufferContentProvider(ByteBuffer byteBuffer, int size, Supplier<Boolean> isOpen) Create a content provider from a byte buffer.
static UploadContentProviders.BaseContentProvider<BufferedInputStream>	fileContentProvider(File file, long offset, int size) Create a content provider from a file.
static UploadContentProviders.BaseContentProvider<BufferedInputStream>	fileContentProvider(File file, long offset, int size, Supplier<Boolean> isOpen) Create a content provider from a file.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

Method Detail

fileContentProvider

public static UploadContentProviders.BaseContentProvider<BufferedInputStream> fileContentProvider(File file,
                                                                                                  long offset,
                                                                                                  int size)

Create a content provider from a file.

Parameters:

file - file to read.

offset - offset in file.

size - of data.

Returns:

the provider

Throws:

IllegalArgumentException - if the offset is negative.

fileContentProvider

public static UploadContentProviders.BaseContentProvider<BufferedInputStream> fileContentProvider(File file,
                                                                                                  long offset,
                                                                                                  int size,
                                                                                                  Supplier<Boolean> isOpen)

Create a content provider from a file.

Parameters:

file - file to read.

offset - offset in file.

size - of data.

isOpen - optional predicate to check if the stream is open.

Returns:

the provider

Throws:

IllegalArgumentException - if the offset is negative.

byteBufferContentProvider

public static UploadContentProviders.BaseContentProvider<org.apache.hadoop.fs.store.ByteBufferInputStream> byteBufferContentProvider(ByteBuffer byteBuffer,
                                                                                                                                     int size)

Create a content provider from a byte buffer. The buffer is not copied and MUST NOT be modified while the upload is taking place.

Parameters:

byteBuffer - buffer to read.

size - size of the data.

Returns:

the provider

Throws:

IllegalArgumentException - if the arguments are invalid.

NullPointerException - if the buffer is null

byteBufferContentProvider

public static UploadContentProviders.BaseContentProvider<org.apache.hadoop.fs.store.ByteBufferInputStream> byteBufferContentProvider(ByteBuffer byteBuffer,
                                                                                                                                     int size,
                                                                                                                                     @Nullable
                                                                                                                                     Supplier<Boolean> isOpen)

Create a content provider from a byte buffer. The buffer is not copied and MUST NOT be modified while the upload is taking place.

Parameters:

byteBuffer - buffer to read.

size - size of the data.

isOpen - optional predicate to check if the stream is open.

Returns:

the provider

Throws:

IllegalArgumentException - if the arguments are invalid.

NullPointerException - if the buffer is null

byteArrayContentProvider

public static UploadContentProviders.BaseContentProvider<ByteArrayInputStream> byteArrayContentProvider(byte[] bytes,
                                                                                                        int offset,
                                                                                                        int size)

Create a content provider for all or part of a byte array. The buffer is not copied and MUST NOT be modified while the upload is taking place.

Parameters:

bytes - buffer to read.

offset - offset in buffer.

size - size of the data.

Returns:

the provider

Throws:

IllegalArgumentException - if the arguments are invalid.

NullPointerException - if the buffer is null.

byteArrayContentProvider

public static UploadContentProviders.BaseContentProvider<ByteArrayInputStream> byteArrayContentProvider(byte[] bytes,
                                                                                                        int offset,
                                                                                                        int size,
                                                                                                        @Nullable
                                                                                                        Supplier<Boolean> isOpen)

Create a content provider for all or part of a byte array. The buffer is not copied and MUST NOT be modified while the upload is taking place.

Parameters:

bytes - buffer to read.

offset - offset in buffer.

size - size of the data.

isOpen - optional predicate to check if the stream is open.

Returns:

the provider

Throws:

IllegalArgumentException - if the arguments are invalid.

NullPointerException - if the buffer is null.

byteArrayContentProvider

public static UploadContentProviders.BaseContentProvider<ByteArrayInputStream> byteArrayContentProvider(byte[] bytes)

Create a content provider for all of a byte array.

Parameters:

bytes - buffer to read.

Returns:

the provider

Throws:

IllegalArgumentException - if the arguments are invalid.

NullPointerException - if the buffer is null.

byteArrayContentProvider

public static UploadContentProviders.BaseContentProvider<ByteArrayInputStream> byteArrayContentProvider(byte[] bytes,
                                                                                                        @Nullable
                                                                                                        Supplier<Boolean> isOpen)

Create a content provider for all of a byte array.

Parameters:

bytes - buffer to read.

isOpen - optional predicate to check if the stream is open.

Returns:

the provider

Throws:

IllegalArgumentException - if the arguments are invalid.

NullPointerException - if the buffer is null.