org.apache.hadoop.fs.s3a.impl.streams

Class AnalyticsStream

java.lang.Object

java.io.InputStream

org.apache.hadoop.fs.FSInputStream

org.apache.hadoop.fs.s3a.impl.streams.ObjectInputStream

org.apache.hadoop.fs.s3a.impl.streams.AnalyticsStream

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.fs.PositionedReadable, org.apache.hadoop.fs.Seekable, org.apache.hadoop.fs.statistics.IOStatisticsSource, org.apache.hadoop.fs.StreamCapabilities

public class AnalyticsStream
extends ObjectInputStream
implements org.apache.hadoop.fs.StreamCapabilities

Analytics stream creates a stream using aws-analytics-accelerator-s3. This stream supports parquet specific optimisations such as parquet-aware prefetching. For more details, see https://github.com/awslabs/analytics-accelerator-s3.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.fs.StreamCapabilities

org.apache.hadoop.fs.StreamCapabilities.StreamCapability

Field Summary

Fields

Modifier and Type	Field and Description
static org.slf4j.Logger	LOG

Fields inherited from interface org.apache.hadoop.fs.StreamCapabilities

ABORTABLE_STREAM, DROPBEHIND, HFLUSH, HSYNC, IOSTATISTICS, IOSTATISTICS_CONTEXT, PREADBYTEBUFFER, READAHEAD, READBYTEBUFFER, UNBUFFER, VECTOREDIO, VECTOREDIO_BUFFERS_SLICED

Constructor Summary

Constructors

Constructor and Description
AnalyticsStream(ObjectReadParameters parameters, software.amazon.s3.analyticsaccelerator.S3SeekableInputStreamFactory s3SeekableInputStreamFactory)

Method Summary

Modifier and Type	Method and Description
protected void	abortInFinalizer() Brute force stream close; invoked by LeakReporter.
int	available()
void	close() Close the stream.
long	getPos()
protected boolean	isClosed()
protected boolean	isStreamOpen() Probe for stream being open.
int	read()
int	read(byte[] buf, int off, int len)
int	readTail(byte[] buf, int off, int len) Reads the last n bytes from the stream into a byte buffer.
void	seek(long pos)
boolean	seekToNewSource(long l)
protected void	throwIfClosed()

Methods inherited from class org.apache.hadoop.fs.s3a.impl.streams.ObjectInputStream

finalize, getBoundedThreadPool, getBucket, getCallbacks, getContentLength, getContext, getInputPolicy, getIOStatistics, getKey, getObjectAttributes, getPathStr, getS3AStreamStatistics, getThreadIOStatistics, getUri, getVectoredIOContext, hasCapability, maxReadSizeForVectorReads, mergeThreadIOStatistics, minSeekForVectorReads, setInputPolicy, streamType, toString

Methods inherited from class org.apache.hadoop.fs.FSInputStream

read, readFully, readFully, validatePositionedReadArgs

Methods inherited from class java.io.InputStream

mark, markSupported, read, reset, skip

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.fs.StreamCapabilities

hasCapability

Methods inherited from interface org.apache.hadoop.fs.PositionedReadable

readVectored, readVectored

Field Detail

LOG

public static final org.slf4j.Logger LOG

Constructor Detail

AnalyticsStream

public AnalyticsStream(ObjectReadParameters parameters,
                       software.amazon.s3.analyticsaccelerator.S3SeekableInputStreamFactory s3SeekableInputStreamFactory)
                throws IOException

Throws:

IOException

Method Detail

read

public int read()
         throws IOException

Specified by:

read in class InputStream

Throws:

IOException

seek

public void seek(long pos)
          throws IOException

Specified by:

seek in interface org.apache.hadoop.fs.Seekable

Specified by:

seek in class org.apache.hadoop.fs.FSInputStream

Throws:

IOException

getPos

public long getPos()

Specified by:

getPos in interface org.apache.hadoop.fs.Seekable

Specified by:

getPos in class org.apache.hadoop.fs.FSInputStream

readTail

public int readTail(byte[] buf,
                    int off,
                    int len)
             throws IOException

Reads the last n bytes from the stream into a byte buffer. Blocks until end of stream is reached. Leaves the position of the stream unaltered.

Parameters:

buf - buffer to read data into

off - start position in buffer at which data is written

len - the number of bytes to read; the n-th byte should be the last byte of the stream.

Returns:

the total number of bytes read into the buffer

Throws:

IOException - if an I/O error occurs

read

public int read(byte[] buf,
                int off,
                int len)
         throws IOException

Overrides:

read in class InputStream

Throws:

IOException

seekToNewSource

public boolean seekToNewSource(long l)
                        throws IOException

Specified by:

seekToNewSource in interface org.apache.hadoop.fs.Seekable

Specified by:

seekToNewSource in class org.apache.hadoop.fs.FSInputStream

Throws:

IOException

available

public int available()
              throws IOException

Overrides:

available in class InputStream

Throws:

IOException

isStreamOpen

protected boolean isStreamOpen()

Description copied from class: ObjectInputStream

Probe for stream being open. Not synchronized; the flag is volatile.

Specified by:

isStreamOpen in class ObjectInputStream

Returns:

true if the stream is still open.

isClosed

protected boolean isClosed()

abortInFinalizer

protected void abortInFinalizer()

Description copied from class: ObjectInputStream

Brute force stream close; invoked by LeakReporter. All exceptions raised are ignored.

Specified by:

abortInFinalizer in class ObjectInputStream

close

public void close()
           throws IOException

Description copied from class: ObjectInputStream

Close the stream. This triggers publishing of the stream statistics back to the filesystem statistics. This operation is synchronized, so that only one thread can attempt to

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class ObjectInputStream

Throws:

IOException - on any problem

throwIfClosed

protected void throwIfClosed()
                      throws IOException

Throws:

IOException