org.apache.hudi.common.table.log.block

Class HoodieDataBlock

java.lang.Object

org.apache.hudi.common.table.log.block.HoodieLogBlock

org.apache.hudi.common.table.log.block.HoodieDataBlock

Direct Known Subclasses:

HoodieAvroDataBlock, HoodieHFileDataBlock, HoodieParquetDataBlock

public abstract class HoodieDataBlock
extends HoodieLogBlock

DataBlock contains a list of records serialized using formats compatible with the base file format. For each base file format there is a corresponding DataBlock format.

The Datablock contains: 1. Data Block version 2. Total number of records in the block 3. Actual serialized content of the records

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hudi.common.table.log.block.HoodieLogBlock

HoodieLogBlock.FooterMetadataType, HoodieLogBlock.HeaderMetadataType, HoodieLogBlock.HoodieLogBlockContentLocation, HoodieLogBlock.HoodieLogBlockType

Field Summary

Fields

Modifier and Type	Field and Description
protected InternalSchema	internalSchema
protected org.apache.avro.Schema	readerSchema

Fields inherited from class org.apache.hudi.common.table.log.block.HoodieLogBlock

readBlockLazily, version

Constructor Summary

Constructors

Modifier	Constructor and Description
	HoodieDataBlock(List<org.apache.avro.generic.IndexedRecord> records, Map<HoodieLogBlock.HeaderMetadataType,String> header, Map<HoodieLogBlock.HeaderMetadataType,String> footer, String keyFieldName) NOTE: This ctor is used on the write-path (ie when records ought to be written into the log)
protected	HoodieDataBlock(Option<byte[]> content, org.apache.hadoop.fs.FSDataInputStream inputStream, boolean readBlockLazily, Option<HoodieLogBlock.HoodieLogBlockContentLocation> blockContentLocation, Option<org.apache.avro.Schema> readerSchema, Map<HoodieLogBlock.HeaderMetadataType,String> headers, Map<HoodieLogBlock.HeaderMetadataType,String> footer, String keyFieldName, boolean enablePointLookups) NOTE: This ctor is used on the write-path (ie when records ought to be written into the log)
protected	HoodieDataBlock(Option<byte[]> content, org.apache.hadoop.fs.FSDataInputStream inputStream, boolean readBlockLazily, Option<HoodieLogBlock.HoodieLogBlockContentLocation> blockContentLocation, Option<org.apache.avro.Schema> readerSchema, Map<HoodieLogBlock.HeaderMetadataType,String> headers, Map<HoodieLogBlock.HeaderMetadataType,String> footer, String keyFieldName, boolean enablePointLookups, InternalSchema internalSchema)

Method Summary

Modifier and Type	Method and Description
protected abstract ClosableIterator<org.apache.avro.generic.IndexedRecord>	deserializeRecords(byte[] content)
abstract HoodieLogBlock.HoodieLogBlockType	getBlockType()
byte[]	getContentBytes()
protected Option<org.apache.avro.Schema.Field>	getKeyField(org.apache.avro.Schema schema)
ClosableIterator<org.apache.avro.generic.IndexedRecord>	getRecordIterator() Returns all the records iterator contained w/in this block.
ClosableIterator<org.apache.avro.generic.IndexedRecord>	getRecordIterator(List<String> keys, boolean fullKey) Batch get of keys of interest.
protected Option<String>	getRecordKey(org.apache.avro.generic.IndexedRecord record)
org.apache.avro.Schema	getSchema()
protected static org.apache.avro.Schema	getWriterSchema(Map<HoodieLogBlock.HeaderMetadataType,String> logBlockHeader)
protected ClosableIterator<org.apache.avro.generic.IndexedRecord>	lookupRecords(List<String> keys, boolean fullKey)
protected ClosableIterator<org.apache.avro.generic.IndexedRecord>	readRecordsFromBlockPayload()
protected abstract byte[]	serializeRecords(List<org.apache.avro.generic.IndexedRecord> records)

Methods inherited from class org.apache.hudi.common.table.log.block.HoodieLogBlock

deflate, getBlockContentLocation, getContent, getLogBlockFooter, getLogBlockHeader, getLogBlockLength, getLogMetadata, getLogMetadataBytes, getMagic, inflate, tryReadContent

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

readerSchema

protected final org.apache.avro.Schema readerSchema

internalSchema

protected InternalSchema internalSchema

Constructor Detail

HoodieDataBlock

public HoodieDataBlock(List<org.apache.avro.generic.IndexedRecord> records,
                       Map<HoodieLogBlock.HeaderMetadataType,String> header,
                       Map<HoodieLogBlock.HeaderMetadataType,String> footer,
                       String keyFieldName)

NOTE: This ctor is used on the write-path (ie when records ought to be written into the log)

HoodieDataBlock

protected HoodieDataBlock(Option<byte[]> content,
                          org.apache.hadoop.fs.FSDataInputStream inputStream,
                          boolean readBlockLazily,
                          Option<HoodieLogBlock.HoodieLogBlockContentLocation> blockContentLocation,
                          Option<org.apache.avro.Schema> readerSchema,
                          Map<HoodieLogBlock.HeaderMetadataType,String> headers,
                          Map<HoodieLogBlock.HeaderMetadataType,String> footer,
                          String keyFieldName,
                          boolean enablePointLookups)

NOTE: This ctor is used on the write-path (ie when records ought to be written into the log)

HoodieDataBlock

protected HoodieDataBlock(Option<byte[]> content,
                          org.apache.hadoop.fs.FSDataInputStream inputStream,
                          boolean readBlockLazily,
                          Option<HoodieLogBlock.HoodieLogBlockContentLocation> blockContentLocation,
                          Option<org.apache.avro.Schema> readerSchema,
                          Map<HoodieLogBlock.HeaderMetadataType,String> headers,
                          Map<HoodieLogBlock.HeaderMetadataType,String> footer,
                          String keyFieldName,
                          boolean enablePointLookups,
                          InternalSchema internalSchema)

Method Detail

getContentBytes

public byte[] getContentBytes()
                       throws IOException

Overrides:

getContentBytes in class HoodieLogBlock

Throws:

IOException

getWriterSchema

protected static org.apache.avro.Schema getWriterSchema(Map<HoodieLogBlock.HeaderMetadataType,String> logBlockHeader)

getRecordIterator

public final ClosableIterator<org.apache.avro.generic.IndexedRecord> getRecordIterator()

Returns all the records iterator contained w/in this block.

getSchema

public org.apache.avro.Schema getSchema()

getRecordIterator

public final ClosableIterator<org.apache.avro.generic.IndexedRecord> getRecordIterator(List<String> keys,
                                                                                       boolean fullKey)
                                                                                throws IOException

Batch get of keys of interest. Implementation can choose to either do full scan and return matched entries or do a seek based parsing and return matched entries.

Parameters:

keys - keys of interest.

Returns:

List of IndexedRecords for the keys of interest.

Throws:

IOException - in case of failures encountered when reading/parsing records

readRecordsFromBlockPayload

protected ClosableIterator<org.apache.avro.generic.IndexedRecord> readRecordsFromBlockPayload()
                                                                                       throws IOException

Throws:

IOException

lookupRecords

protected ClosableIterator<org.apache.avro.generic.IndexedRecord> lookupRecords(List<String> keys,
                                                                                boolean fullKey)
                                                                         throws IOException

Throws:

IOException

serializeRecords

protected abstract byte[] serializeRecords(List<org.apache.avro.generic.IndexedRecord> records)
                                    throws IOException

Throws:

IOException

deserializeRecords

protected abstract ClosableIterator<org.apache.avro.generic.IndexedRecord> deserializeRecords(byte[] content)
                                                                                       throws IOException

Throws:

IOException

getBlockType

public abstract HoodieLogBlock.HoodieLogBlockType getBlockType()

Specified by:

getBlockType in class HoodieLogBlock

getKeyField

protected Option<org.apache.avro.Schema.Field> getKeyField(org.apache.avro.Schema schema)

getRecordKey

protected Option<String> getRecordKey(org.apache.avro.generic.IndexedRecord record)