org.apache.hudi.common.table.log

Class AbstractHoodieLogRecordReader

java.lang.Object

org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader

Direct Known Subclasses:

HoodieMergedLogRecordScanner, HoodieUnMergedLogRecordScanner

public abstract class AbstractHoodieLogRecordReader
extends Object

Implements logic to scan log blocks and expose valid and deleted log records to subclass implementation. Subclass is free to either apply merging or expose raw data back to the caller. NOTE: If readBlockLazily is turned on, does not merge, instead keeps reading log blocks and merges everything at once This is an optimization to avoid seek() back and forth to read new block (forward seek()) and lazily read content of seen block (reverse and forward seek()) during merge | | Read Block 1 Metadata | | Read Block 1 Data | | | Read Block 2 Metadata | | Read Block 2 Data | | I/O Pass 1 | ..................... | I/O Pass 2 | ................. | | | Read Block N Metadata | | Read Block N Data |

This results in two I/O passes over the log file.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	AbstractHoodieLogRecordReader.Builder Builder used to build AbstractHoodieLogRecordScanner.
protected static class	AbstractHoodieLogRecordReader.KeySpec

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	forceFullScan
protected List<String>	logFilePaths
protected org.apache.avro.Schema	readerSchema

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	AbstractHoodieLogRecordReader(org.apache.hadoop.fs.FileSystem fs, String basePath, List<String> logFilePaths, org.apache.avro.Schema readerSchema, String latestInstantTime, boolean readBlocksLazily, boolean reverseReader, int bufferSize, Option<InstantRange> instantRange, boolean withOperationField)
protected	AbstractHoodieLogRecordReader(org.apache.hadoop.fs.FileSystem fs, String basePath, List<String> logFilePaths, org.apache.avro.Schema readerSchema, String latestInstantTime, boolean readBlocksLazily, boolean reverseReader, int bufferSize, Option<InstantRange> instantRange, boolean withOperationField, boolean forceFullScan, Option<String> partitionName, InternalSchema internalSchema)

Method Summary

Modifier and Type	Method and Description
protected HoodieAvroRecord<?>	createHoodieRecord(org.apache.avro.generic.IndexedRecord rec, HoodieTableConfig hoodieTableConfig, String payloadClassFQN, String preCombineField, boolean withOperationField, Option<Pair<String,String>> simpleKeyGenFields, Option<String> partitionName) Create @HoodieRecord from the @IndexedRecord.
protected String	getKeyField()
Option<String>	getPartitionName()
protected String	getPayloadClassFQN()
float	getProgress() Return progress of scanning as a float between 0.0 to 1.0.
long	getTotalCorruptBlocks()
long	getTotalLogBlocks()
long	getTotalLogFiles()
long	getTotalLogRecords()
long	getTotalRollbacks()
boolean	isWithOperationField()
protected abstract void	processNextDeletedRecord(DeleteRecord deleteRecord) Process next deleted record.
protected abstract void	processNextRecord(HoodieRecord<? extends HoodieRecordPayload> hoodieRecord) Process next record.
void	scan()
void	scan(List<String> keys)
protected void	scanInternal(Option<AbstractHoodieLogRecordReader.KeySpec> keySpecOpt)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

readerSchema

protected final org.apache.avro.Schema readerSchema

logFilePaths

protected final List<String> logFilePaths

forceFullScan

protected final boolean forceFullScan

Constructor Detail

AbstractHoodieLogRecordReader

protected AbstractHoodieLogRecordReader(org.apache.hadoop.fs.FileSystem fs,
                                        String basePath,
                                        List<String> logFilePaths,
                                        org.apache.avro.Schema readerSchema,
                                        String latestInstantTime,
                                        boolean readBlocksLazily,
                                        boolean reverseReader,
                                        int bufferSize,
                                        Option<InstantRange> instantRange,
                                        boolean withOperationField)

AbstractHoodieLogRecordReader

protected AbstractHoodieLogRecordReader(org.apache.hadoop.fs.FileSystem fs,
                                        String basePath,
                                        List<String> logFilePaths,
                                        org.apache.avro.Schema readerSchema,
                                        String latestInstantTime,
                                        boolean readBlocksLazily,
                                        boolean reverseReader,
                                        int bufferSize,
                                        Option<InstantRange> instantRange,
                                        boolean withOperationField,
                                        boolean forceFullScan,
                                        Option<String> partitionName,
                                        InternalSchema internalSchema)

Method Detail

getKeyField

protected String getKeyField()

scan

public void scan()

scan

public void scan(List<String> keys)

scanInternal

protected void scanInternal(Option<AbstractHoodieLogRecordReader.KeySpec> keySpecOpt)

createHoodieRecord

protected HoodieAvroRecord<?> createHoodieRecord(org.apache.avro.generic.IndexedRecord rec,
                                                 HoodieTableConfig hoodieTableConfig,
                                                 String payloadClassFQN,
                                                 String preCombineField,
                                                 boolean withOperationField,
                                                 Option<Pair<String,String>> simpleKeyGenFields,
                                                 Option<String> partitionName)

Create @HoodieRecord from the @IndexedRecord.

Parameters:

rec - - IndexedRecord to create the HoodieRecord from

hoodieTableConfig - - Table config

payloadClassFQN - - Payload class fully qualified name

preCombineField - - PreCombine field

withOperationField - - Whether operation field is enabled

simpleKeyGenFields - - Key generator fields when populate meta fields is tuened off

partitionName - - Partition name

Returns:

HoodieRecord created from the IndexedRecord

processNextRecord

protected abstract void processNextRecord(HoodieRecord<? extends HoodieRecordPayload> hoodieRecord)
                                   throws Exception

Process next record.

Parameters:

hoodieRecord - Hoodie Record to process

Throws:

Exception

processNextDeletedRecord

protected abstract void processNextDeletedRecord(DeleteRecord deleteRecord)

Process next deleted record.

Parameters:

deleteRecord - Deleted record(hoodie key and ordering value)

getProgress

public float getProgress()

Return progress of scanning as a float between 0.0 to 1.0.

getTotalLogFiles

public long getTotalLogFiles()

getTotalLogRecords

public long getTotalLogRecords()

getTotalLogBlocks

public long getTotalLogBlocks()

getPayloadClassFQN

protected String getPayloadClassFQN()

getPartitionName

public Option<String> getPartitionName()

getTotalRollbacks

public long getTotalRollbacks()

getTotalCorruptBlocks

public long getTotalCorruptBlocks()

isWithOperationField

public boolean isWithOperationField()