org.apache.hadoop.fs.s3a.impl.streams

Class ObjectInputStream

java.lang.Object

java.io.InputStream

org.apache.hadoop.fs.FSInputStream

org.apache.hadoop.fs.s3a.impl.streams.ObjectInputStream

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.fs.PositionedReadable, org.apache.hadoop.fs.Seekable, org.apache.hadoop.fs.statistics.IOStatisticsSource, org.apache.hadoop.fs.StreamCapabilities

Direct Known Subclasses:

AnalyticsStream, S3AInputStream, S3APrefetchingInputStream

public abstract class ObjectInputStream
extends org.apache.hadoop.fs.FSInputStream
implements org.apache.hadoop.fs.StreamCapabilities, org.apache.hadoop.fs.statistics.IOStatisticsSource

A stream of data from an S3 object.

The base class includes common methods, stores common data and incorporates leak tracking.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.fs.StreamCapabilities

org.apache.hadoop.fs.StreamCapabilities.StreamCapability

Field Summary

Fields inherited from interface org.apache.hadoop.fs.StreamCapabilities

ABORTABLE_STREAM, DROPBEHIND, HFLUSH, HSYNC, IOSTATISTICS, IOSTATISTICS_CONTEXT, PREADBYTEBUFFER, READAHEAD, READBYTEBUFFER, UNBUFFER, VECTOREDIO, VECTOREDIO_BUFFERS_SLICED

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	ObjectInputStream(InputStreamType streamType, ObjectReadParameters parameters) Constructor.

Method Summary

Modifier and Type	Method and Description
protected abstract void	abortInFinalizer() Brute force stream close; invoked by LeakReporter.
void	close() Close the stream.
protected void	finalize() Finalizer.
protected ExecutorService	getBoundedThreadPool() Thread pool used for bounded IO operations.
protected String	getBucket() Store bucket.
protected ObjectInputStreamCallbacks	getCallbacks() Callbacks for reading input stream data from the S3 Store.
protected long	getContentLength() Content length from HEAD or openFile option.
protected S3AReadOpContext	getContext() Read-specific operation context structure.
S3AInputPolicy	getInputPolicy() Get the current input policy.
org.apache.hadoop.fs.statistics.IOStatistics	getIOStatistics()
protected String	getKey() Store key.
protected S3ObjectAttributes	getObjectAttributes() Attributes of the remote object.
protected String	getPathStr() Path URI as a string.
S3AInputStreamStatistics	getS3AStreamStatistics() Access the input stream statistics.
protected org.apache.hadoop.fs.statistics.IOStatisticsAggregator	getThreadIOStatistics() Aggregator used to aggregate per thread IOStatistics.
protected String	getUri() URI of path.
protected VectoredIOContext	getVectoredIOContext() Get Vectored IO context.
boolean	hasCapability(String capability) Declare the base capabilities implemented by this class and so by all subclasses.
protected abstract boolean	isStreamOpen() Probe for stream being open.
int	maxReadSizeForVectorReads() .
protected void	mergeThreadIOStatistics(org.apache.hadoop.fs.statistics.IOStatistics streamIOStats) Merging the current thread's IOStatistics with the current IOStatistics context.
int	minSeekForVectorReads() .
protected void	setInputPolicy(S3AInputPolicy inputPolicy) Set/update the input policy of the stream.
InputStreamType	streamType()
String	toString()

Methods inherited from class org.apache.hadoop.fs.FSInputStream

getPos, read, readFully, readFully, seek, seekToNewSource, validatePositionedReadArgs

Methods inherited from class java.io.InputStream

available, mark, markSupported, read, read, read, reset, skip

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.fs.PositionedReadable

readVectored, readVectored

Constructor Detail

ObjectInputStream

protected ObjectInputStream(InputStreamType streamType,
                            ObjectReadParameters parameters)

Constructor.

Parameters:

streamType - stream type enum.

parameters - extensible parameter list.

Method Detail

isStreamOpen

protected abstract boolean isStreamOpen()

Probe for stream being open. Not synchronized; the flag is volatile.

Returns:

true if the stream is still open.

abortInFinalizer

protected abstract void abortInFinalizer()

Brute force stream close; invoked by LeakReporter. All exceptions raised are ignored.

close

public void close()
           throws IOException

Close the stream. This triggers publishing of the stream statistics back to the filesystem statistics. This operation is synchronized, so that only one thread can attempt to

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class InputStream

Throws:

IOException - on any problem

mergeThreadIOStatistics

protected void mergeThreadIOStatistics(org.apache.hadoop.fs.statistics.IOStatistics streamIOStats)

Merging the current thread's IOStatistics with the current IOStatistics context.

Parameters:

streamIOStats - Stream statistics to be merged into thread statistics aggregator.

finalize

protected void finalize()
                 throws Throwable

Finalizer.

Verify that the inner stream is closed.

If it is not, it means streams are being leaked in application code. Log a warning, including the stack trace of the caller, then abort the stream.

This does not attempt to invoke close() as that is a more complex operation, and this method is being executed during a GC finalization phase.

Applications MUST close their streams; this is a defensive operation to return http connections and warn the end users that their applications are at risk of running out of connections.

Overrides:

finalize in class Object

Throws:

Throwable

getInputPolicy

@VisibleForTesting
public S3AInputPolicy getInputPolicy()

Get the current input policy.

Returns:

input policy.

setInputPolicy

protected void setInputPolicy(S3AInputPolicy inputPolicy)

Set/update the input policy of the stream. This updates the stream statistics.

Parameters:

inputPolicy - new input policy.

getS3AStreamStatistics

@InterfaceAudience.Private
 @InterfaceStability.Unstable
 @VisibleForTesting
public S3AInputStreamStatistics getS3AStreamStatistics()

Access the input stream statistics. This is for internal testing and may be removed without warning.

Returns:

the statistics for this input stream

getIOStatistics

public org.apache.hadoop.fs.statistics.IOStatistics getIOStatistics()

Specified by:

getIOStatistics in interface org.apache.hadoop.fs.statistics.IOStatisticsSource

hasCapability

public boolean hasCapability(String capability)

Declare the base capabilities implemented by this class and so by all subclasses.

Subclasses MUST override this if they add more capabilities, or actually remove any of these.

Specified by:

hasCapability in interface org.apache.hadoop.fs.StreamCapabilities

Parameters:

capability - string to query the stream support for.

Returns:

true if all implementations are known to have the specific capability.

getContext

protected final S3AReadOpContext getContext()

Read-specific operation context structure.

Returns:

Read-specific operation context structure.

getCallbacks

protected final ObjectInputStreamCallbacks getCallbacks()

Callbacks for reading input stream data from the S3 Store.

Returns:

Callbacks for reading input stream data from the S3 Store.

getBoundedThreadPool

protected final ExecutorService getBoundedThreadPool()

Thread pool used for bounded IO operations.

Returns:

Thread pool used for bounded IO operations.

getUri

protected final String getUri()

URI of path.

Returns:

URI of path.

getBucket

protected final String getBucket()

Store bucket.

Returns:

Store bucket.

getKey

protected final String getKey()

Store key.

Returns:

Store key.

getPathStr

protected final String getPathStr()

Path URI as a string.

Returns:

Path URI as a string.

getContentLength

protected final long getContentLength()

Content length from HEAD or openFile option.

Returns:

Content length from HEAD or openFile option.

getThreadIOStatistics

protected final org.apache.hadoop.fs.statistics.IOStatisticsAggregator getThreadIOStatistics()

Aggregator used to aggregate per thread IOStatistics.

Returns:

Aggregator used to aggregate per thread IOStatistics.

getObjectAttributes

protected final S3ObjectAttributes getObjectAttributes()

Attributes of the remote object.

Returns:

Attributes of the remote object.

getVectoredIOContext

protected VectoredIOContext getVectoredIOContext()

Get Vectored IO context.

Returns:

Vectored IO context.

minSeekForVectorReads

public int minSeekForVectorReads()

.

Specified by:

minSeekForVectorReads in interface org.apache.hadoop.fs.PositionedReadable

maxReadSizeForVectorReads

public int maxReadSizeForVectorReads()

.

Specified by:

maxReadSizeForVectorReads in interface org.apache.hadoop.fs.PositionedReadable

streamType

public InputStreamType streamType()

toString

public String toString()

Overrides:

toString in class org.apache.hadoop.fs.FSInputStream