org.apache.hadoop.fs.s3a

Class S3AInputStream

java.lang.Object

java.io.InputStream

org.apache.hadoop.fs.FSInputStream

org.apache.hadoop.fs.s3a.impl.streams.ObjectInputStream

org.apache.hadoop.fs.s3a.S3AInputStream

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.fs.CanSetReadahead, org.apache.hadoop.fs.CanUnbuffer, org.apache.hadoop.fs.PositionedReadable, org.apache.hadoop.fs.Seekable, org.apache.hadoop.fs.statistics.IOStatisticsSource, org.apache.hadoop.fs.StreamCapabilities

@InterfaceAudience.Private
 @InterfaceStability.Evolving
public class S3AInputStream
extends ObjectInputStream
implements org.apache.hadoop.fs.CanSetReadahead, org.apache.hadoop.fs.CanUnbuffer, org.apache.hadoop.fs.StreamCapabilities, org.apache.hadoop.fs.statistics.IOStatisticsSource

The input stream for an S3A object. As this stream seeks withing an object, it may close then re-open the stream. When this happens, any updated stream data may be retrieved, and, given the consistency model of Amazon S3, outdated data may in fact be picked up. As a result, the outcome of reading from a stream of an object which is actively manipulated during the read process is "undefined". The class is marked as private as code should not be creating instances themselves. Any extra feature (e.g instrumentation) should be considered unstable. Because it prints some of the state of the instrumentation, the output of toString() must also be considered unstable.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.fs.StreamCapabilities

org.apache.hadoop.fs.StreamCapabilities.StreamCapability

Field Summary

Fields

Modifier and Type	Field and Description
static String	E_NEGATIVE_READAHEAD_VALUE
static String	OPERATION_OPEN
static String	OPERATION_REOPEN

Fields inherited from interface org.apache.hadoop.fs.StreamCapabilities

ABORTABLE_STREAM, DROPBEHIND, HFLUSH, HSYNC, IOSTATISTICS, IOSTATISTICS_CONTEXT, PREADBYTEBUFFER, READAHEAD, READBYTEBUFFER, UNBUFFER, VECTOREDIO, VECTOREDIO_BUFFERS_SLICED

Constructor Summary

Constructors

Constructor and Description
S3AInputStream(ObjectReadParameters parameters) Create the stream.

Method Summary

Modifier and Type	Method and Description
protected void	abortInFinalizer() Brute force stream close; invoked by LeakReporter.
int	available()
void	close() Close the stream.
long	getContentRangeFinish()
long	getContentRangeStart()
long	getPos()
long	getReadahead() Get the current readahead value.
software.amazon.awssdk.core.ResponseInputStream<software.amazon.awssdk.services.s3.model.GetObjectResponse>	getWrappedStream() Get the wrapped stream.
boolean	hasCapability(String capability) Declare the base capabilities implemented by this class and so by all subclasses.
boolean	isObjectStreamOpen() Is the inner object stream open?
protected boolean	isStreamOpen() Probe for stream being open.
boolean	markSupported()
int	read()
int	read(byte[] buf, int off, int len) This updates the statistics on read operations started and whether or not the read operation "completed", that is: returned the exact number of bytes requested.
void	readFully(long position, byte[] buffer, int offset, int length) Subclass readFully() operation which only seeks at the start of the series of operations; seeking back at the end.
void	readVectored(List<? extends org.apache.hadoop.fs.FileRange> ranges, IntFunction<ByteBuffer> allocate) Pass to readVectored(List, IntFunction, Consumer) with the VectoredReadUtils.LOG_BYTE_BUFFER_RELEASED releaser.
void	readVectored(List<? extends org.apache.hadoop.fs.FileRange> ranges, IntFunction<ByteBuffer> allocate, Consumer<ByteBuffer> release) Vectored read implementation for S3AInputStream.
long	remainingInCurrentRequest() Bytes left in the current request.
long	remainingInFile() Bytes left in stream.
boolean	resetConnection() Forcibly reset the stream, by aborting the connection.
void	seek(long targetPos)
boolean	seekToNewSource(long targetPos)
void	setReadahead(Long readahead)
String	toString() String value includes statistics as well as stream state.
void	unbuffer() Closes the underlying S3 stream, and merges the ObjectInputStream.streamStatistics instance associated with the stream.
static long	validateReadahead(Long readahead) from a possibly null Long value, return a valid readahead.

Methods inherited from class org.apache.hadoop.fs.s3a.impl.streams.ObjectInputStream

finalize, getBoundedThreadPool, getBucket, getCallbacks, getContentLength, getContext, getInputPolicy, getIOStatistics, getKey, getObjectAttributes, getPathStr, getS3AStreamStatistics, getThreadIOStatistics, getUri, getVectoredIOContext, maxReadSizeForVectorReads, mergeThreadIOStatistics, minSeekForVectorReads, setInputPolicy, streamType

Methods inherited from class org.apache.hadoop.fs.FSInputStream

read, readFully, validatePositionedReadArgs

Methods inherited from class java.io.InputStream

mark, read, reset, skip

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.fs.statistics.IOStatisticsSource

getIOStatistics

Field Detail

E_NEGATIVE_READAHEAD_VALUE

public static final String E_NEGATIVE_READAHEAD_VALUE

See Also:

Constant Field Values

OPERATION_OPEN

public static final String OPERATION_OPEN

See Also:

Constant Field Values

OPERATION_REOPEN

public static final String OPERATION_REOPEN

See Also:

Constant Field Values

Constructor Detail

S3AInputStream

public S3AInputStream(ObjectReadParameters parameters)

Create the stream. This does not attempt to open it; that is only done on the first actual read() operation.

Parameters:

parameters - creation parameters.

Method Detail

isStreamOpen

protected boolean isStreamOpen()

Probe for stream being open. Not synchronized; the flag is volatile.

Specified by:

isStreamOpen in class ObjectInputStream

Returns:

true if the stream is still open.

abortInFinalizer

protected void abortInFinalizer()

Brute force stream close; invoked by LeakReporter. All exceptions raised are ignored.

Specified by:

abortInFinalizer in class ObjectInputStream

getPos

public long getPos()
            throws IOException

Specified by:

getPos in interface org.apache.hadoop.fs.Seekable

Specified by:

getPos in class org.apache.hadoop.fs.FSInputStream

Throws:

IOException

seek

public void seek(long targetPos)
          throws IOException

Specified by:

seek in interface org.apache.hadoop.fs.Seekable

Specified by:

seek in class org.apache.hadoop.fs.FSInputStream

Throws:

IOException

seekToNewSource

public boolean seekToNewSource(long targetPos)
                        throws IOException

Specified by:

seekToNewSource in interface org.apache.hadoop.fs.Seekable

Specified by:

seekToNewSource in class org.apache.hadoop.fs.FSInputStream

Throws:

IOException

read

@Retries.RetryTranslated
public int read()
                                  throws IOException

Specified by:

read in class InputStream

Throws:

IOException

read

@Retries.RetryTranslated
public int read(byte[] buf,
                                         int off,
                                         int len)
                                  throws IOException

This updates the statistics on read operations started and whether or not the read operation "completed", that is: returned the exact number of bytes requested.

Overrides:

read in class InputStream

Throws:

IOException - if there are other problems

close

public void close()
           throws IOException

Close the stream. This triggers publishing of the stream statistics back to the filesystem statistics. This operation is synchronized, so that only one thread can attempt to close the connection; all later/blocked calls are no-ops.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class ObjectInputStream

Throws:

IOException - on any problem

resetConnection

@InterfaceStability.Unstable
public boolean resetConnection()
                                                     throws IOException

Forcibly reset the stream, by aborting the connection. The next read() operation will trigger the opening of a new HTTPS connection. This is potentially very inefficient, and should only be invoked in extreme circumstances. It logs at info for this reason. Blocks until the abort is completed.

Returns:

true if the connection was actually reset.

Throws:

IOException - if invoked on a closed stream.

available

public int available()
              throws IOException

Overrides:

available in class InputStream

Throws:

IOException

remainingInFile

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public long remainingInFile()

Bytes left in stream.

Returns:

how many bytes are left to read

remainingInCurrentRequest

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public long remainingInCurrentRequest()

Bytes left in the current request. Only valid if there is an active request.

Returns:

how many bytes are left to read in the current GET.

getContentRangeFinish

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public long getContentRangeFinish()

getContentRangeStart

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public long getContentRangeStart()

markSupported

public boolean markSupported()

Overrides:

markSupported in class InputStream

toString

@InterfaceStability.Unstable
public String toString()

String value includes statistics as well as stream state. Important: there are no guarantees as to the stability of this value.

Overrides:

toString in class ObjectInputStream

Returns:

a string value for printing in logs/diagnostics

readFully

@Retries.RetryTranslated
public void readFully(long position,
                                               byte[] buffer,
                                               int offset,
                                               int length)
                                        throws IOException

Subclass readFully() operation which only seeks at the start of the series of operations; seeking back at the end. This is significantly higher performance if multiple read attempts are needed to fetch the data, as it does not break the HTTP connection. To maintain thread safety requirements, this operation is synchronized for the duration of the sequence.

Specified by:

readFully in interface org.apache.hadoop.fs.PositionedReadable

Overrides:

readFully in class org.apache.hadoop.fs.FSInputStream

Throws:

IOException

readVectored

public void readVectored(List<? extends org.apache.hadoop.fs.FileRange> ranges,
                         IntFunction<ByteBuffer> allocate)
                  throws IOException

Pass to readVectored(List, IntFunction, Consumer) with the VectoredReadUtils.LOG_BYTE_BUFFER_RELEASED releaser.

Specified by:

readVectored in interface org.apache.hadoop.fs.PositionedReadable

Parameters:

ranges - the byte ranges to read.

allocate - the function to allocate ByteBuffer.

Throws:

IOException - IOE if any.

readVectored

public void readVectored(List<? extends org.apache.hadoop.fs.FileRange> ranges,
                         IntFunction<ByteBuffer> allocate,
                         Consumer<ByteBuffer> release)
                  throws IOException

Vectored read implementation for S3AInputStream.

Specified by:

readVectored in interface org.apache.hadoop.fs.PositionedReadable

Parameters:

ranges - the byte ranges to read.

allocate - the function to allocate ByteBuffer.

release - the function to release a ByteBuffer.

Throws:

IOException - IOE if any.

setReadahead

public void setReadahead(Long readahead)

Specified by:

setReadahead in interface org.apache.hadoop.fs.CanSetReadahead

getReadahead

public long getReadahead()

Get the current readahead value.

Returns:

a non-negative readahead value

validateReadahead

public static long validateReadahead(@Nullable
                                     Long readahead)

from a possibly null Long value, return a valid readahead.

Parameters:

readahead - new readahead

Returns:

a natural number.

Throws:

IllegalArgumentException - if the range is invalid.

unbuffer

public void unbuffer()

Closes the underlying S3 stream, and merges the ObjectInputStream.streamStatistics instance associated with the stream. Also sets the stopVectoredIOOperations flag to true such that active vectored read operations are terminated. However termination of old vectored reads are not guaranteed if a new vectored read operation is initiated after unbuffer is called.

Specified by:

unbuffer in interface org.apache.hadoop.fs.CanUnbuffer

hasCapability

public boolean hasCapability(String capability)

Description copied from class: ObjectInputStream

Declare the base capabilities implemented by this class and so by all subclasses.

Subclasses MUST override this if they add more capabilities, or actually remove any of these.

Specified by:

hasCapability in interface org.apache.hadoop.fs.StreamCapabilities

Overrides:

hasCapability in class ObjectInputStream

Parameters:

capability - string to query the stream support for.

Returns:

true if all implementations are known to have the specific capability.

isObjectStreamOpen

@VisibleForTesting
public boolean isObjectStreamOpen()

Is the inner object stream open?

Returns:

true if there is an active HTTP request to S3.

getWrappedStream

@VisibleForTesting
public software.amazon.awssdk.core.ResponseInputStream<software.amazon.awssdk.services.s3.model.GetObjectResponse> getWrappedStream()

Get the wrapped stream. This is for testing only.

Returns:

the wrapped stream, or null if there is none.