org.apache.hadoop.fs.s3a.prefetch

Class S3APrefetchingInputStream

java.lang.Object

java.io.InputStream

org.apache.hadoop.fs.FSInputStream

org.apache.hadoop.fs.s3a.prefetch.S3APrefetchingInputStream

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.fs.CanSetReadahead, org.apache.hadoop.fs.PositionedReadable, org.apache.hadoop.fs.Seekable, org.apache.hadoop.fs.statistics.IOStatisticsSource, org.apache.hadoop.fs.StreamCapabilities

public class S3APrefetchingInputStream
extends org.apache.hadoop.fs.FSInputStream
implements org.apache.hadoop.fs.CanSetReadahead, org.apache.hadoop.fs.StreamCapabilities, org.apache.hadoop.fs.statistics.IOStatisticsSource

Enhanced InputStream for reading from S3. This implementation provides improved read throughput by asynchronously prefetching blocks of configurable size from the underlying S3 file.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.fs.StreamCapabilities

org.apache.hadoop.fs.StreamCapabilities.StreamCapability

Field Summary

Fields inherited from interface org.apache.hadoop.fs.StreamCapabilities

ABORTABLE_STREAM, DROPBEHIND, HFLUSH, HSYNC, IOSTATISTICS, IOSTATISTICS_CONTEXT, PREADBYTEBUFFER, READAHEAD, READBYTEBUFFER, UNBUFFER, VECTOREDIO

Constructor Summary

Constructors

Constructor and Description
S3APrefetchingInputStream(S3AReadOpContext context, S3ObjectAttributes s3Attributes, S3AInputStream.InputStreamCallbacks client, S3AInputStreamStatistics streamStatistics, org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.LocalDirAllocator localDirAllocator) Initializes a new instance of the S3APrefetchingInputStream class.

Method Summary

Modifier and Type	Method and Description
int	available() Returns the number of bytes available for reading without blocking.
void	close() Closes this stream and releases all acquired resources.
org.apache.hadoop.fs.statistics.IOStatistics	getIOStatistics() Gets the internal IO statistics.
long	getPos() Gets the current position.
S3AInputStreamStatistics	getS3AStreamStatistics() Access the input stream statistics.
boolean	hasCapability(String capability) Indicates whether the given capability is supported by this stream.
protected boolean	isClosed()
boolean	markSupported()
int	read() Reads and returns one byte from this stream.
int	read(byte[] buffer, int offset, int len) Reads up to len bytes from this stream and copies them into the given buffer starting at the given offset.
void	seek(long pos) Updates internal data such that the next read will take place at the given pos.
boolean	seekToNewSource(long targetPos)
void	setReadahead(Long readahead) Sets the number of bytes to read ahead each time.
protected void	throwIfClosed()

Methods inherited from class org.apache.hadoop.fs.FSInputStream

read, readFully, readFully, toString, validatePositionedReadArgs

Methods inherited from class java.io.InputStream

mark, read, reset, skip

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.fs.PositionedReadable

maxReadSizeForVectorReads, minSeekForVectorReads, readVectored

Constructor Detail

S3APrefetchingInputStream

public S3APrefetchingInputStream(S3AReadOpContext context,
                                 S3ObjectAttributes s3Attributes,
                                 S3AInputStream.InputStreamCallbacks client,
                                 S3AInputStreamStatistics streamStatistics,
                                 org.apache.hadoop.conf.Configuration conf,
                                 org.apache.hadoop.fs.LocalDirAllocator localDirAllocator)

Initializes a new instance of the S3APrefetchingInputStream class.

Parameters:

context - read-specific operation context.

s3Attributes - attributes of the S3 object being read.

client - callbacks used for interacting with the underlying S3 client.

streamStatistics - statistics for this stream.

conf - the configuration.

localDirAllocator - the local dir allocator instance retrieved from S3A FS.

Throws:

IllegalArgumentException - if context is null.

IllegalArgumentException - if s3Attributes is null.

IllegalArgumentException - if client is null.

Method Detail

available

public int available()
              throws IOException

Returns the number of bytes available for reading without blocking.

Overrides:

available in class InputStream

Returns:

the number of bytes available for reading without blocking.

Throws:

IOException - if there is an IO error during this operation.

getPos

public long getPos()
            throws IOException

Gets the current position. If the underlying S3 input stream is closed, it returns last read current position from the underlying steam. If the current position was never read and the underlying input stream is closed, this would return 0.

Specified by:

getPos in interface org.apache.hadoop.fs.Seekable

Specified by:

getPos in class org.apache.hadoop.fs.FSInputStream

Returns:

the current position.

Throws:

IOException - if there is an IO error during this operation.

read

public int read()
         throws IOException

Reads and returns one byte from this stream.

Specified by:

read in class InputStream

Returns:

the next byte from this stream.

Throws:

IOException - if there is an IO error during this operation.

read

public int read(byte[] buffer,
                int offset,
                int len)
         throws IOException

Reads up to len bytes from this stream and copies them into the given buffer starting at the given offset. Returns the number of bytes actually copied in to the given buffer.

Overrides:

read in class InputStream

Parameters:

buffer - the buffer to copy data into.

offset - data is copied starting at this offset.

len - max number of bytes to copy.

Returns:

the number of bytes actually copied in to the given buffer.

Throws:

IOException - if there is an IO error during this operation.

close

public void close()
           throws IOException

Closes this stream and releases all acquired resources.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class InputStream

Throws:

IOException - if there is an IO error during this operation.

seek

public void seek(long pos)
          throws IOException

Updates internal data such that the next read will take place at the given pos.

Specified by:

seek in interface org.apache.hadoop.fs.Seekable

Specified by:

seek in class org.apache.hadoop.fs.FSInputStream

Parameters:

pos - new read position.

Throws:

IOException - if there is an IO error during this operation.

setReadahead

public void setReadahead(Long readahead)

Sets the number of bytes to read ahead each time.

Specified by:

setReadahead in interface org.apache.hadoop.fs.CanSetReadahead

Parameters:

readahead - the number of bytes to read ahead each time..

hasCapability

public boolean hasCapability(String capability)

Indicates whether the given capability is supported by this stream.

Specified by:

hasCapability in interface org.apache.hadoop.fs.StreamCapabilities

Parameters:

capability - the capability to check.

Returns:

true if the given capability is supported by this stream, false otherwise.

getS3AStreamStatistics

@InterfaceAudience.Private
 @InterfaceStability.Unstable
 @VisibleForTesting
public S3AInputStreamStatistics getS3AStreamStatistics()

Access the input stream statistics. This is for internal testing and may be removed without warning.

Returns:

the statistics for this input stream

getIOStatistics

public org.apache.hadoop.fs.statistics.IOStatistics getIOStatistics()

Gets the internal IO statistics.

Specified by:

getIOStatistics in interface org.apache.hadoop.fs.statistics.IOStatisticsSource

Returns:

the internal IO statistics.

isClosed

protected boolean isClosed()

throwIfClosed

protected void throwIfClosed()
                      throws IOException

Throws:

IOException

seekToNewSource

public boolean seekToNewSource(long targetPos)
                        throws IOException

Specified by:

seekToNewSource in interface org.apache.hadoop.fs.Seekable

Specified by:

seekToNewSource in class org.apache.hadoop.fs.FSInputStream

Throws:

IOException

markSupported

public boolean markSupported()

Overrides:

markSupported in class InputStream