org.apache.hudi.table.format.cow.vector.reader

Class AbstractColumnReader<V extends org.apache.flink.table.data.columnar.vector.writable.WritableColumnVector>

java.lang.Object

org.apache.hudi.table.format.cow.vector.reader.AbstractColumnReader<V>

All Implemented Interfaces:

org.apache.flink.formats.parquet.vector.reader.ColumnReader<V>

Direct Known Subclasses:

FixedLenBytesColumnReader, Int64TimestampColumnReader

public abstract class AbstractColumnReader<V extends org.apache.flink.table.data.columnar.vector.writable.WritableColumnVector>
extends Object
implements org.apache.flink.formats.parquet.vector.reader.ColumnReader<V>

Abstract ColumnReader. See org.apache.parquet.column.impl.ColumnReaderImpl, part of the code is referred from Apache Spark and Apache Parquet.

Note: Reference Flink release 1.11.2 AbstractColumnReader because some of the package scope methods.

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.parquet.column.ColumnDescriptor	descriptor
protected org.apache.parquet.column.Dictionary	dictionary The dictionary, if this column has dictionary encoding.
protected int	maxDefLevel Maximum definition level for this column.
protected org.apache.hudi.table.format.cow.vector.reader.RunLengthDecoder	runLenDecoder Run length decoder for data and dictionary.

Constructor Summary

Constructors

Constructor and Description
AbstractColumnReader(org.apache.parquet.column.ColumnDescriptor descriptor, org.apache.parquet.column.page.PageReader pageReader)

Method Summary

Modifier and Type	Method and Description
protected void	afterReadPage() After read a page, we may need some initialization.
protected void	checkTypeName(org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName expectedName)
protected abstract void	readBatch(int rowId, int num, V column) Read batch from runLenDecoder and dataInputStream.
protected abstract void	readBatchFromDictionaryIds(int rowId, int num, V column, org.apache.flink.table.data.columnar.vector.writable.WritableIntVector dictionaryIds) Decode dictionary ids to data.
void	readToVector(int readNumber, V vector) Reads `total` values from this columnReader into column.
protected boolean	supportLazyDecode() Support lazy dictionary ids decode.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

dictionary

protected final org.apache.parquet.column.Dictionary dictionary

The dictionary, if this column has dictionary encoding.

maxDefLevel

protected final int maxDefLevel

Maximum definition level for this column.

descriptor

protected final org.apache.parquet.column.ColumnDescriptor descriptor

runLenDecoder

protected org.apache.hudi.table.format.cow.vector.reader.RunLengthDecoder runLenDecoder

Run length decoder for data and dictionary.

Constructor Detail

AbstractColumnReader

public AbstractColumnReader(org.apache.parquet.column.ColumnDescriptor descriptor,
                            org.apache.parquet.column.page.PageReader pageReader)
                     throws IOException

Throws:

IOException

Method Detail

checkTypeName

protected void checkTypeName(org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName expectedName)

readToVector

public final void readToVector(int readNumber,
                               V vector)
                        throws IOException

Reads `total` values from this columnReader into column.

Specified by:

readToVector in interface org.apache.flink.formats.parquet.vector.reader.ColumnReader<V extends org.apache.flink.table.data.columnar.vector.writable.WritableColumnVector>

Throws:

IOException

afterReadPage

protected void afterReadPage()

After read a page, we may need some initialization.

supportLazyDecode

protected boolean supportLazyDecode()

Support lazy dictionary ids decode. See more in ParquetDictionary. If return false, we will decode all the data first.

readBatch

protected abstract void readBatch(int rowId,
                                  int num,
                                  V column)

Read batch from runLenDecoder and dataInputStream.

readBatchFromDictionaryIds

protected abstract void readBatchFromDictionaryIds(int rowId,
                                                   int num,
                                                   V column,
                                                   org.apache.flink.table.data.columnar.vector.writable.WritableIntVector dictionaryIds)

Decode dictionary ids to data. From runLenDecoder and dictionaryIdsDecoder.