org.apache.hadoop.fs.s3a.prefetch

Class S3APrefetchingInputStream

java.lang.Object

java.io.InputStream

org.apache.hadoop.fs.FSInputStream

org.apache.hadoop.fs.s3a.impl.streams.ObjectInputStream

org.apache.hadoop.fs.s3a.prefetch.S3APrefetchingInputStream

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.fs.CanSetReadahead, org.apache.hadoop.fs.PositionedReadable, org.apache.hadoop.fs.Seekable, org.apache.hadoop.fs.statistics.IOStatisticsSource, org.apache.hadoop.fs.StreamCapabilities

public class S3APrefetchingInputStream
extends ObjectInputStream
implements org.apache.hadoop.fs.CanSetReadahead, org.apache.hadoop.fs.StreamCapabilities, org.apache.hadoop.fs.statistics.IOStatisticsSource

Enhanced InputStream for reading from S3. This implementation provides improved read throughput by asynchronously prefetching blocks of configurable size from the underlying S3 file.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.fs.StreamCapabilities

org.apache.hadoop.fs.StreamCapabilities.StreamCapability

Field Summary

Fields inherited from interface org.apache.hadoop.fs.StreamCapabilities

ABORTABLE_STREAM, DROPBEHIND, HFLUSH, HSYNC, IOSTATISTICS, IOSTATISTICS_CONTEXT, PREADBYTEBUFFER, READAHEAD, READBYTEBUFFER, UNBUFFER, VECTOREDIO, VECTOREDIO_BUFFERS_SLICED

Constructor Summary

Constructors

Constructor and Description
S3APrefetchingInputStream(ObjectReadParameters parameters, org.apache.hadoop.conf.Configuration conf, PrefetchOptions prefetchOptions) Initializes a new instance of the S3APrefetchingInputStream class.

Method Summary

Modifier and Type	Method and Description
protected void	abortInFinalizer() Brute force stream close; invoked by LeakReporter.
int	available() Returns the number of bytes available for reading without blocking.
void	close() Closes this stream and releases all acquired resources.
org.apache.hadoop.fs.statistics.IOStatistics	getIOStatistics() Gets the internal IO statistics.
long	getPos() Gets the current position.
S3AInputStreamStatistics	getS3AStreamStatistics() Access the input stream statistics.
boolean	hasCapability(String capability) Indicates whether the given capability is supported by this stream.
protected boolean	isClosed()
protected boolean	isStreamOpen() Probe for stream being open.
boolean	markSupported()
int	read() Reads and returns one byte from this stream.
int	read(byte[] buffer, int offset, int len) Reads up to len bytes from this stream and copies them into the given buffer starting at the given offset.
void	seek(long pos) Updates internal data such that the next read will take place at the given pos.
boolean	seekToNewSource(long targetPos)
void	setReadahead(Long readahead) Sets the number of bytes to read ahead each time.
protected void	throwIfClosed()

Methods inherited from class org.apache.hadoop.fs.s3a.impl.streams.ObjectInputStream

finalize, getBoundedThreadPool, getBucket, getCallbacks, getContentLength, getContext, getInputPolicy, getKey, getObjectAttributes, getPathStr, getThreadIOStatistics, getUri, getVectoredIOContext, maxReadSizeForVectorReads, mergeThreadIOStatistics, minSeekForVectorReads, setInputPolicy, streamType, toString

Methods inherited from class org.apache.hadoop.fs.FSInputStream

read, readFully, readFully, validatePositionedReadArgs

Methods inherited from class java.io.InputStream

mark, read, reset, skip

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.fs.PositionedReadable

readVectored, readVectored

Constructor Detail

S3APrefetchingInputStream

public S3APrefetchingInputStream(ObjectReadParameters parameters,
                                 org.apache.hadoop.conf.Configuration conf,
                                 PrefetchOptions prefetchOptions)

Initializes a new instance of the S3APrefetchingInputStream class.

Parameters:

parameters - creation parameters.

conf - the configuration.

prefetchOptions - prefetch stream specific options

Throws:

NullPointerException - if a required parameter is null.

Method Detail

available

public int available()
              throws IOException

Returns the number of bytes available for reading without blocking.

Overrides:

available in class InputStream

Returns:

the number of bytes available for reading without blocking.

Throws:

IOException - if there is an IO error during this operation.

getPos

public long getPos()
            throws IOException

Gets the current position. If the underlying S3 input stream is closed, it returns last read current position from the underlying steam. If the current position was never read and the underlying input stream is closed, this would return 0.

Specified by:

getPos in interface org.apache.hadoop.fs.Seekable

Specified by:

getPos in class org.apache.hadoop.fs.FSInputStream

Returns:

the current position.

Throws:

IOException - if there is an IO error during this operation.

read

public int read()
         throws IOException

Reads and returns one byte from this stream.

Specified by:

read in class InputStream

Returns:

the next byte from this stream.

Throws:

IOException - if there is an IO error during this operation.

read

public int read(byte[] buffer,
                int offset,
                int len)
         throws IOException

Reads up to len bytes from this stream and copies them into the given buffer starting at the given offset. Returns the number of bytes actually copied in to the given buffer.

Overrides:

read in class InputStream

Parameters:

buffer - the buffer to copy data into.

offset - data is copied starting at this offset.

len - max number of bytes to copy.

Returns:

the number of bytes actually copied in to the given buffer.

Throws:

IOException - if there is an IO error during this operation.

close

public void close()
           throws IOException

Closes this stream and releases all acquired resources.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class ObjectInputStream

Throws:

IOException - if there is an IO error during this operation.

isStreamOpen

protected boolean isStreamOpen()

Description copied from class: ObjectInputStream

Probe for stream being open. Not synchronized; the flag is volatile.

Specified by:

isStreamOpen in class ObjectInputStream

Returns:

true if the stream is still open.

abortInFinalizer

protected void abortInFinalizer()

Description copied from class: ObjectInputStream

Brute force stream close; invoked by LeakReporter. All exceptions raised are ignored.

Specified by:

abortInFinalizer in class ObjectInputStream

seek

public void seek(long pos)
          throws IOException

Updates internal data such that the next read will take place at the given pos.

Specified by:

seek in interface org.apache.hadoop.fs.Seekable

Specified by:

seek in class org.apache.hadoop.fs.FSInputStream

Parameters:

pos - new read position.

Throws:

IOException - if there is an IO error during this operation.

setReadahead

public void setReadahead(Long readahead)

Sets the number of bytes to read ahead each time.

Specified by:

setReadahead in interface org.apache.hadoop.fs.CanSetReadahead

Parameters:

readahead - the number of bytes to read ahead each time..

hasCapability

public boolean hasCapability(String capability)

Indicates whether the given capability is supported by this stream.

Specified by:

hasCapability in interface org.apache.hadoop.fs.StreamCapabilities

Overrides:

hasCapability in class ObjectInputStream

Parameters:

capability - the capability to check.

Returns:

true if the given capability is supported by this stream, false otherwise.

getS3AStreamStatistics

@InterfaceAudience.Private
 @InterfaceStability.Unstable
 @VisibleForTesting
public S3AInputStreamStatistics getS3AStreamStatistics()

Access the input stream statistics. This is for internal testing and may be removed without warning.

Overrides:

getS3AStreamStatistics in class ObjectInputStream

Returns:

the statistics for this input stream

getIOStatistics

public org.apache.hadoop.fs.statistics.IOStatistics getIOStatistics()

Gets the internal IO statistics.

Specified by:

getIOStatistics in interface org.apache.hadoop.fs.statistics.IOStatisticsSource

Overrides:

getIOStatistics in class ObjectInputStream

Returns:

the internal IO statistics.

isClosed

protected boolean isClosed()

throwIfClosed

protected void throwIfClosed()
                      throws IOException

Throws:

IOException

seekToNewSource

public boolean seekToNewSource(long targetPos)
                        throws IOException

Specified by:

seekToNewSource in interface org.apache.hadoop.fs.Seekable

Specified by:

seekToNewSource in class org.apache.hadoop.fs.FSInputStream

Throws:

IOException

markSupported

public boolean markSupported()

Overrides:

markSupported in class InputStream