org.apache.hudi.common.table.log.block

Class HoodieParquetDataBlock

java.lang.Object

org.apache.hudi.common.table.log.block.HoodieLogBlock

org.apache.hudi.common.table.log.block.HoodieDataBlock

org.apache.hudi.common.table.log.block.HoodieParquetDataBlock

public class HoodieParquetDataBlock
extends HoodieDataBlock

HoodieParquetDataBlock contains a list of records serialized using Parquet.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hudi.common.table.log.block.HoodieLogBlock

HoodieLogBlock.FooterMetadataType, HoodieLogBlock.HeaderMetadataType, HoodieLogBlock.HoodieLogBlockContentLocation, HoodieLogBlock.HoodieLogBlockType

Field Summary

Fields inherited from class org.apache.hudi.common.table.log.block.HoodieDataBlock

internalSchema, readerSchema

Fields inherited from class org.apache.hudi.common.table.log.block.HoodieLogBlock

readBlockLazily, version

Constructor Summary

Constructors

Constructor and Description
HoodieParquetDataBlock(org.apache.hadoop.fs.FSDataInputStream inputStream, Option<byte[]> content, boolean readBlockLazily, HoodieLogBlock.HoodieLogBlockContentLocation logBlockContentLocation, Option<org.apache.avro.Schema> readerSchema, Map<HoodieLogBlock.HeaderMetadataType,String> header, Map<HoodieLogBlock.HeaderMetadataType,String> footer, String keyField)
HoodieParquetDataBlock(List<org.apache.avro.generic.IndexedRecord> records, Map<HoodieLogBlock.HeaderMetadataType,String> header, String keyField, org.apache.parquet.hadoop.metadata.CompressionCodecName compressionCodecName)

Method Summary

Modifier and Type	Method and Description
protected ClosableIterator<org.apache.avro.generic.IndexedRecord>	deserializeRecords(byte[] content)
HoodieLogBlock.HoodieLogBlockType	getBlockType()
static ClosableIterator<org.apache.avro.generic.IndexedRecord>	getProjectedParquetRecordsIterator(org.apache.hadoop.conf.Configuration conf, org.apache.avro.Schema readerSchema, org.apache.parquet.io.InputFile inputFile)
protected ClosableIterator<org.apache.avro.generic.IndexedRecord>	readRecordsFromBlockPayload() NOTE: We're overriding the whole reading sequence to make sure we properly respect the requested Reader's schema and only fetch the columns that have been explicitly requested by the caller (providing projected Reader's schema)
protected byte[]	serializeRecords(List<org.apache.avro.generic.IndexedRecord> records)

Methods inherited from class org.apache.hudi.common.table.log.block.HoodieDataBlock

getContentBytes, getKeyField, getRecordIterator, getRecordIterator, getRecordKey, getSchema, getWriterSchema, lookupRecords

Methods inherited from class org.apache.hudi.common.table.log.block.HoodieLogBlock

deflate, getBlockContentLocation, getContent, getLogBlockFooter, getLogBlockHeader, getLogBlockLength, getLogMetadata, getLogMetadataBytes, getMagic, inflate, tryReadContent

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HoodieParquetDataBlock

public HoodieParquetDataBlock(org.apache.hadoop.fs.FSDataInputStream inputStream,
                              Option<byte[]> content,
                              boolean readBlockLazily,
                              HoodieLogBlock.HoodieLogBlockContentLocation logBlockContentLocation,
                              Option<org.apache.avro.Schema> readerSchema,
                              Map<HoodieLogBlock.HeaderMetadataType,String> header,
                              Map<HoodieLogBlock.HeaderMetadataType,String> footer,
                              String keyField)

HoodieParquetDataBlock

public HoodieParquetDataBlock(@Nonnull
                              List<org.apache.avro.generic.IndexedRecord> records,
                              @Nonnull
                              Map<HoodieLogBlock.HeaderMetadataType,String> header,
                              @Nonnull
                              String keyField,
                              @Nonnull
                              org.apache.parquet.hadoop.metadata.CompressionCodecName compressionCodecName)

Method Detail

getBlockType

public HoodieLogBlock.HoodieLogBlockType getBlockType()

Specified by:

getBlockType in class HoodieDataBlock

serializeRecords

protected byte[] serializeRecords(List<org.apache.avro.generic.IndexedRecord> records)
                           throws IOException

Specified by:

serializeRecords in class HoodieDataBlock

Throws:

IOException

getProjectedParquetRecordsIterator

public static ClosableIterator<org.apache.avro.generic.IndexedRecord> getProjectedParquetRecordsIterator(org.apache.hadoop.conf.Configuration conf,
                                                                                                         org.apache.avro.Schema readerSchema,
                                                                                                         org.apache.parquet.io.InputFile inputFile)
                                                                                                  throws IOException

Throws:

IOException

readRecordsFromBlockPayload

protected ClosableIterator<org.apache.avro.generic.IndexedRecord> readRecordsFromBlockPayload()
                                                                                       throws IOException

NOTE: We're overriding the whole reading sequence to make sure we properly respect the requested Reader's schema and only fetch the columns that have been explicitly requested by the caller (providing projected Reader's schema)

Overrides:

readRecordsFromBlockPayload in class HoodieDataBlock

Throws:

IOException

deserializeRecords

protected ClosableIterator<org.apache.avro.generic.IndexedRecord> deserializeRecords(byte[] content)

Specified by:

deserializeRecords in class HoodieDataBlock