org.apache.hadoop.fs.s3a.prefetch

Class S3ARemoteInputStream

java.lang.Object

java.io.InputStream

org.apache.hadoop.fs.s3a.prefetch.S3ARemoteInputStream

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.fs.CanSetReadahead, org.apache.hadoop.fs.statistics.IOStatisticsSource, org.apache.hadoop.fs.StreamCapabilities

Direct Known Subclasses:

S3ACachingInputStream, S3AInMemoryInputStream

public abstract class S3ARemoteInputStream
extends InputStream
implements org.apache.hadoop.fs.CanSetReadahead, org.apache.hadoop.fs.StreamCapabilities, org.apache.hadoop.fs.statistics.IOStatisticsSource

Provides an InputStream that allows reading from an S3 file.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.fs.StreamCapabilities

org.apache.hadoop.fs.StreamCapabilities.StreamCapability

Field Summary

Fields inherited from interface org.apache.hadoop.fs.StreamCapabilities

ABORTABLE_STREAM, DROPBEHIND, HFLUSH, HSYNC, IOSTATISTICS, IOSTATISTICS_CONTEXT, PREADBYTEBUFFER, READAHEAD, READBYTEBUFFER, UNBUFFER, VECTOREDIO

Constructor Summary

Constructors

Constructor and Description
S3ARemoteInputStream(S3AReadOpContext context, S3ObjectAttributes s3Attributes, S3AInputStream.InputStreamCallbacks client, S3AInputStreamStatistics streamStatistics) Initializes a new instance of the S3ARemoteInputStream class.

Method Summary

Modifier and Type	Method and Description
int	available() Returns the number of bytes that can read from this stream without blocking.
void	close() Closes this stream and releases all acquired resources.
protected abstract boolean	ensureCurrentBuffer() Ensures that a non-empty valid buffer is available for immediate reading.
protected org.apache.hadoop.fs.impl.prefetch.BlockData	getBlockData()
protected S3AReadOpContext	getContext()
protected S3ARemoteObject	getFile()
protected org.apache.hadoop.fs.impl.prefetch.FilePosition	getFilePosition()
org.apache.hadoop.fs.statistics.IOStatistics	getIOStatistics() Gets the internal IO statistics.
protected String	getName()
protected long	getNextReadPos()
protected String	getOffsetStr(long offset)
long	getPos() Gets the current position.
protected S3ARemoteObjectReader	getReader()
S3AInputStreamStatistics	getS3AStreamStatistics() Access the input stream statistics.
protected S3ARemoteObject	getS3File()
protected S3ObjectAttributes	getS3ObjectAttributes()
boolean	hasCapability(String capability) Indicates whether the given capability is supported by this stream.
protected boolean	isClosed()
void	mark(int readlimit)
boolean	markSupported()
int	read()
int	read(byte[] buffer) Reads bytes from this stream and copies them into the given buffer starting at the beginning (offset 0).
int	read(byte[] buffer, int offset, int len) Reads up to len bytes from this stream and copies them into the given buffer starting at the given offset.
void	reset()
void	seek(long pos) Moves the current read position so that the next read will occur at pos.
void	setReadahead(Long readahead) Sets the number of bytes to read ahead each time.
long	skip(long n)
protected void	throwIfClosed()
protected void	throwIfInvalidSeek(long pos)
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

S3ARemoteInputStream

public S3ARemoteInputStream(S3AReadOpContext context,
                            S3ObjectAttributes s3Attributes,
                            S3AInputStream.InputStreamCallbacks client,
                            S3AInputStreamStatistics streamStatistics)

Initializes a new instance of the S3ARemoteInputStream class.

Parameters:

context - read-specific operation context.

s3Attributes - attributes of the S3 object being read.

client - callbacks used for interacting with the underlying S3 client.

streamStatistics - statistics for this stream.

Throws:

IllegalArgumentException - if context is null.

IllegalArgumentException - if s3Attributes is null.

IllegalArgumentException - if client is null.

Method Detail

getIOStatistics

public org.apache.hadoop.fs.statistics.IOStatistics getIOStatistics()

Gets the internal IO statistics.

Specified by:

getIOStatistics in interface org.apache.hadoop.fs.statistics.IOStatisticsSource

Returns:

the internal IO statistics.

getS3AStreamStatistics

@InterfaceAudience.Private
 @InterfaceStability.Unstable
public S3AInputStreamStatistics getS3AStreamStatistics()

Access the input stream statistics. This is for internal testing and may be removed without warning.

Returns:

the statistics for this input stream

setReadahead

public void setReadahead(Long readahead)

Sets the number of bytes to read ahead each time.

Specified by:

setReadahead in interface org.apache.hadoop.fs.CanSetReadahead

Parameters:

readahead - the number of bytes to read ahead each time..

hasCapability

public boolean hasCapability(String capability)

Indicates whether the given capability is supported by this stream.

Specified by:

hasCapability in interface org.apache.hadoop.fs.StreamCapabilities

Parameters:

capability - the capability to check.

Returns:

true if the given capability is supported by this stream, false otherwise.

available

public int available()
              throws IOException

Returns the number of bytes that can read from this stream without blocking.

Overrides:

available in class InputStream

Throws:

IOException

getPos

public long getPos()
            throws IOException

Gets the current position.

Returns:

the current position.

Throws:

IOException - if there is an IO error during this operation.

seek

public void seek(long pos)
          throws IOException

Moves the current read position so that the next read will occur at pos.

Parameters:

pos - the absolute position to seek to.

Throws:

IOException - if there an error during this operation.

IllegalArgumentException - if pos is outside of the range [0, file size].

ensureCurrentBuffer

protected abstract boolean ensureCurrentBuffer()
                                        throws IOException

Ensures that a non-empty valid buffer is available for immediate reading. It returns true when at least one such buffer is available for reading. It returns false on reaching the end of the stream.

Returns:

true if at least one such buffer is available for reading, false otherwise.

Throws:

IOException - if there is an IO error during this operation.

read

public int read()
         throws IOException

Specified by:

read in class InputStream

Throws:

IOException

read

public int read(byte[] buffer)
         throws IOException

Reads bytes from this stream and copies them into the given buffer starting at the beginning (offset 0). Returns the number of bytes actually copied in to the given buffer.

Overrides:

read in class InputStream

Parameters:

buffer - the buffer to copy data into.

Returns:

the number of bytes actually copied in to the given buffer.

Throws:

IOException - if there is an IO error during this operation.

read

public int read(byte[] buffer,
                int offset,
                int len)
         throws IOException

Reads up to len bytes from this stream and copies them into the given buffer starting at the given offset. Returns the number of bytes actually copied in to the given buffer.

Overrides:

read in class InputStream

Parameters:

buffer - the buffer to copy data into.

offset - data is copied starting at this offset.

len - max number of bytes to copy.

Returns:

the number of bytes actually copied in to the given buffer.

Throws:

IOException - if there is an IO error during this operation.

getFile

protected S3ARemoteObject getFile()

getReader

protected S3ARemoteObjectReader getReader()

getS3ObjectAttributes

protected S3ObjectAttributes getS3ObjectAttributes()

getFilePosition

protected org.apache.hadoop.fs.impl.prefetch.FilePosition getFilePosition()

getName

protected String getName()

isClosed

protected boolean isClosed()

getNextReadPos

protected long getNextReadPos()

getBlockData

protected org.apache.hadoop.fs.impl.prefetch.BlockData getBlockData()

getContext

protected S3AReadOpContext getContext()

getS3File

protected S3ARemoteObject getS3File()

getOffsetStr

protected String getOffsetStr(long offset)

close

public void close()
           throws IOException

Closes this stream and releases all acquired resources.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Overrides:

close in class InputStream

Throws:

IOException - if there is an IO error during this operation.

markSupported

public boolean markSupported()

Overrides:

markSupported in class InputStream

toString

public String toString()

Overrides:

toString in class Object

throwIfClosed

protected void throwIfClosed()
                      throws IOException

Throws:

IOException

throwIfInvalidSeek

protected void throwIfInvalidSeek(long pos)
                           throws EOFException

Throws:

EOFException

mark

public void mark(int readlimit)

Overrides:

mark in class InputStream

reset

public void reset()

Overrides:

reset in class InputStream

skip

public long skip(long n)

Overrides:

skip in class InputStream