org.apache.hadoop.fs.s3a.prefetch

Class S3ACachingInputStream

java.lang.Object

java.io.InputStream

org.apache.hadoop.fs.s3a.prefetch.S3ARemoteInputStream

org.apache.hadoop.fs.s3a.prefetch.S3ACachingInputStream

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.fs.CanSetReadahead, org.apache.hadoop.fs.statistics.IOStatisticsSource, org.apache.hadoop.fs.StreamCapabilities

public class S3ACachingInputStream
extends S3ARemoteInputStream

Provides an InputStream that allows reading from an S3 file. Prefetched blocks are cached to local disk if a seek away from the current block is issued.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.fs.StreamCapabilities

org.apache.hadoop.fs.StreamCapabilities.StreamCapability

Field Summary

Fields inherited from interface org.apache.hadoop.fs.StreamCapabilities

ABORTABLE_STREAM, DROPBEHIND, HFLUSH, HSYNC, IOSTATISTICS, IOSTATISTICS_CONTEXT, PREADBYTEBUFFER, READAHEAD, READBYTEBUFFER, UNBUFFER, VECTOREDIO

Constructor Summary

Constructors

Constructor and Description
S3ACachingInputStream(S3AReadOpContext context, S3ObjectAttributes s3Attributes, S3AInputStream.InputStreamCallbacks client, S3AInputStreamStatistics streamStatistics, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.LocalDirAllocator localDirAllocator) Initializes a new instance of the S3ACachingInputStream class.

Method Summary

Modifier and Type	Method and Description
void	close() Closes this stream and releases all acquired resources.
protected org.apache.hadoop.fs.impl.prefetch.BlockManager	createBlockManager(org.apache.hadoop.fs.impl.prefetch.BlockManagerParameters blockManagerParameters, S3ARemoteObjectReader reader)
protected boolean	ensureCurrentBuffer() Ensures that a non-empty valid buffer is available for immediate reading.
String	toString()

Methods inherited from class org.apache.hadoop.fs.s3a.prefetch.S3ARemoteInputStream

available, getBlockData, getContext, getFile, getFilePosition, getIOStatistics, getName, getNextReadPos, getOffsetStr, getPos, getReader, getS3AStreamStatistics, getS3File, getS3ObjectAttributes, hasCapability, isClosed, mark, markSupported, read, read, read, reset, seek, setReadahead, skip, throwIfClosed, throwIfInvalidSeek

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

S3ACachingInputStream

public S3ACachingInputStream(S3AReadOpContext context,
                             S3ObjectAttributes s3Attributes,
                             S3AInputStream.InputStreamCallbacks client,
                             S3AInputStreamStatistics streamStatistics,
                             org.apache.hadoop.conf.Configuration conf,
                             org.apache.hadoop.fs.LocalDirAllocator localDirAllocator)

Initializes a new instance of the S3ACachingInputStream class.

Parameters:

context - read-specific operation context.

s3Attributes - attributes of the S3 object being read.

client - callbacks used for interacting with the underlying S3 client.

streamStatistics - statistics for this stream.

conf - the configuration.

localDirAllocator - the local dir allocator instance.

Throws:

IllegalArgumentException - if context is null.

IllegalArgumentException - if s3Attributes is null.

IllegalArgumentException - if client is null.

Method Detail

close

public void close()
           throws IOException

Description copied from class: S3ARemoteInputStream

Closes this stream and releases all acquired resources.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class S3ARemoteInputStream

Throws:

IOException - if there is an IO error during this operation.

ensureCurrentBuffer

protected boolean ensureCurrentBuffer()
                               throws IOException

Description copied from class: S3ARemoteInputStream

Ensures that a non-empty valid buffer is available for immediate reading. It returns true when at least one such buffer is available for reading. It returns false on reaching the end of the stream.

Specified by:

ensureCurrentBuffer in class S3ARemoteInputStream

Returns:

true if at least one such buffer is available for reading, false otherwise.

Throws:

IOException - if there is an IO error during this operation.

toString

public String toString()

Overrides:

toString in class S3ARemoteInputStream

createBlockManager

protected org.apache.hadoop.fs.impl.prefetch.BlockManager createBlockManager(@Nonnull
                                                                             org.apache.hadoop.fs.impl.prefetch.BlockManagerParameters blockManagerParameters,
                                                                             S3ARemoteObjectReader reader)