org.apache.iceberg.spark.data.vectorized

Class ColumnarBatchReader

java.lang.Object

org.apache.iceberg.arrow.vectorized.BaseBatchReader<org.apache.spark.sql.vectorized.ColumnarBatch>

org.apache.iceberg.spark.data.vectorized.ColumnarBatchReader

All Implemented Interfaces:

org.apache.iceberg.parquet.VectorizedReader<org.apache.spark.sql.vectorized.ColumnarBatch>

public class ColumnarBatchReader
extends org.apache.iceberg.arrow.vectorized.BaseBatchReader<org.apache.spark.sql.vectorized.ColumnarBatch>

VectorizedReader that returns Spark's ColumnarBatch to support Spark's vectorized read path. The ColumnarBatch returned is created by passing in the Arrow vectors populated via delegated read calls to VectorReader(s).

Field Summary

Fields inherited from class org.apache.iceberg.arrow.vectorized.BaseBatchReader

readers, vectorHolders

Constructor Summary

Constructors

Constructor and Description
ColumnarBatchReader(java.util.List<org.apache.iceberg.parquet.VectorizedReader<?>> readers)

Method Summary

Modifier and Type	Method and Description
org.apache.spark.sql.vectorized.ColumnarBatch	read(org.apache.spark.sql.vectorized.ColumnarBatch reuse, int numRowsToRead)
void	setDeleteFilter(org.apache.iceberg.data.DeleteFilter<org.apache.spark.sql.catalyst.InternalRow> deleteFilter)
void	setRowGroupInfo(org.apache.parquet.column.page.PageReadStore pageStore, java.util.Map<org.apache.parquet.hadoop.metadata.ColumnPath,org.apache.parquet.hadoop.metadata.ColumnChunkMetaData> metaData, long rowPosition)

Methods inherited from class org.apache.iceberg.arrow.vectorized.BaseBatchReader

close, closeVectors, setBatchSize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ColumnarBatchReader

public ColumnarBatchReader(java.util.List<org.apache.iceberg.parquet.VectorizedReader<?>> readers)

Method Detail

setRowGroupInfo

public void setRowGroupInfo(org.apache.parquet.column.page.PageReadStore pageStore,
                            java.util.Map<org.apache.parquet.hadoop.metadata.ColumnPath,org.apache.parquet.hadoop.metadata.ColumnChunkMetaData> metaData,
                            long rowPosition)

Specified by:

setRowGroupInfo in interface org.apache.iceberg.parquet.VectorizedReader<org.apache.spark.sql.vectorized.ColumnarBatch>

Overrides:

setRowGroupInfo in class org.apache.iceberg.arrow.vectorized.BaseBatchReader<org.apache.spark.sql.vectorized.ColumnarBatch>

setDeleteFilter

public void setDeleteFilter(org.apache.iceberg.data.DeleteFilter<org.apache.spark.sql.catalyst.InternalRow> deleteFilter)

read

public final org.apache.spark.sql.vectorized.ColumnarBatch read(org.apache.spark.sql.vectorized.ColumnarBatch reuse,
                                                                int numRowsToRead)