org.apache.iceberg.spark.data

Class SparkOrcReader

java.lang.Object

org.apache.iceberg.spark.data.SparkOrcReader

All Implemented Interfaces:

org.apache.iceberg.orc.OrcRowReader<org.apache.spark.sql.catalyst.InternalRow>

public class SparkOrcReader
extends java.lang.Object
implements org.apache.iceberg.orc.OrcRowReader<org.apache.spark.sql.catalyst.InternalRow>

Converts the OrcIterator, which returns ORC's VectorizedRowBatch to a set of Spark's UnsafeRows.

It minimizes allocations by reusing most of the objects in the implementation.

Constructor Summary

Constructors

Constructor and Description
SparkOrcReader(org.apache.iceberg.Schema expectedSchema, org.apache.orc.TypeDescription readSchema)
SparkOrcReader(org.apache.iceberg.Schema expectedSchema, org.apache.orc.TypeDescription readOrcSchema, java.util.Map<java.lang.Integer,?> idToConstant)

Method Summary

Modifier and Type	Method and Description
org.apache.spark.sql.catalyst.InternalRow	read(org.apache.orc.storage.ql.exec.vector.VectorizedRowBatch batch, int row)
void	setBatchContext(long batchOffsetInFile)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkOrcReader

public SparkOrcReader(org.apache.iceberg.Schema expectedSchema,
                      org.apache.orc.TypeDescription readSchema)

SparkOrcReader

public SparkOrcReader(org.apache.iceberg.Schema expectedSchema,
                      org.apache.orc.TypeDescription readOrcSchema,
                      java.util.Map<java.lang.Integer,?> idToConstant)

Method Detail

read

public org.apache.spark.sql.catalyst.InternalRow read(org.apache.orc.storage.ql.exec.vector.VectorizedRowBatch batch,
                                                      int row)

Specified by:

read in interface org.apache.iceberg.orc.OrcRowReader<org.apache.spark.sql.catalyst.InternalRow>

setBatchContext

public void setBatchContext(long batchOffsetInFile)

Specified by:

setBatchContext in interface org.apache.iceberg.orc.OrcRowReader<org.apache.spark.sql.catalyst.InternalRow>