org.apache.parquet.hadoop.util.wrapped.io

Class VectorIoBridge

java.lang.Object

org.apache.parquet.hadoop.util.wrapped.io.VectorIoBridge

public final class VectorIoBridge
extends Object

Vector IO bridge.

This loads the PositionedReadable method:

   void readVectored(List[?extends FileRange] ranges,
     IntFunction[ByteBuffer] allocate) throws IOException
 

It is made slightly easier because of type erasure; the signature of the function is actually Void.class method(List.class, IntFunction.class).

There are some counters to aid in testing; the toString() method will print them and the loaded method, for use in tests and debug logs.

Method Summary

Modifier and Type	Method and Description
boolean	available() Is the bridge available?
static VectorIoBridge	availableInstance() Get the instance verifying that the API is available.
static boolean	bridgeAvailable() Is the bridge available for use?
long	getBlocksRead() How many blocks were read?
long	getBytesRead() How many bytes of data have been read.
long	getVectorReads() How many vector read calls have been made?
boolean	hasCapability(org.apache.hadoop.fs.FSDataInputStream stream, String capability) Does a stream implement StreamCapability and, if so, is the capability available.
static VectorIoBridge	instance() Get the singleton instance.
boolean	readVectoredAvailable(org.apache.hadoop.fs.FSDataInputStream stream, ByteBufferAllocator allocator) Is the vectored IO API available for the given stream and allocator in this Hadoop runtime?
void	readVectoredRanges(org.apache.hadoop.fs.FSDataInputStream stream, List<ParquetFileRange> ranges, ByteBufferAllocator allocator) Read fully a list of file ranges asynchronously from this file.
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Method Detail

readVectoredAvailable

public boolean readVectoredAvailable(org.apache.hadoop.fs.FSDataInputStream stream,
                                     ByteBufferAllocator allocator)

Is the vectored IO API available for the given stream and allocator in this Hadoop runtime?

Parameters:

stream - input stream to query.

allocator - allocator to be used.

Returns:

true if the stream declares the capability is available.

available

public boolean available()

Is the bridge available?

Returns:

true if readVectored() is available.

readVectoredRanges

public void readVectoredRanges(org.apache.hadoop.fs.FSDataInputStream stream,
                               List<ParquetFileRange> ranges,
                               ByteBufferAllocator allocator)
                        throws IOException

Read fully a list of file ranges asynchronously from this file. The default iterates through the ranges to read each synchronously, but the intent is that FSDataInputStream subclasses can make more efficient readers. The ParquetFileRange parameters all have their data read futures set to the range reads of the associated operations; callers must await these to complete.

As a result of the call, each range will have FileRange.setData(CompletableFuture) called with a future that when complete will have a ByteBuffer with the data from the file's range.

The position returned by getPos() after readVectored() is undefined.

If a file is changed while the readVectored() operation is in progress, the output is undefined. Some ranges may have old data, some may have new and some may have both.

While a readVectored() operation is in progress, normal read api calls may block.

Parameters:

stream - stream from where the data has to be read.

ranges - parquet file ranges.

allocator - buffer allocator.

Throws:

UnsupportedOperationException - if the API is not available.

EOFException - if a range is past the known end of the file.

IOException - any IO problem initiating the read operations.

IllegalArgumentException - if there are overlapping ranges or a range element is invalid

toString

public String toString()

Overrides:

toString in class Object

hasCapability

public boolean hasCapability(org.apache.hadoop.fs.FSDataInputStream stream,
                             String capability)

Does a stream implement StreamCapability and, if so, is the capability available. The call will return false if

1. The method is not found
2. The method is found but throws an exception when invoked
3. The method is found, invoked and returns false

Put differently: it will only return true if the method probe returned true.

Parameters:

stream - input stream to query.

capability - the capability to look for.

Returns:

true if the stream declares the capability is available.

getVectorReads

public long getVectorReads()

How many vector read calls have been made?

Returns:

the count of vector reads.

getBlocksRead

public long getBlocksRead()

How many blocks were read?

Returns:

the count of blocks read.

getBytesRead

public long getBytesRead()

How many bytes of data have been read.

Returns:

the count of bytes read.

instance

public static VectorIoBridge instance()

Get the singleton instance.

Returns:

instance.

bridgeAvailable

public static boolean bridgeAvailable()

Is the bridge available for use?

Returns:

true if the vector IO API is available.

availableInstance

public static VectorIoBridge availableInstance()

Get the instance verifying that the API is available.

Returns:

an available instance, always

Throws:

UnsupportedOperationException - if it is not.