org.apache.hudi.common.table.log

Class AbstractHoodieLogRecordReader

java.lang.Object

org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader

Direct Known Subclasses:

HoodieMergedLogRecordScanner, HoodieUnMergedLogRecordScanner

public abstract class AbstractHoodieLogRecordReader
extends Object

Implements logic to scan log blocks and expose valid and deleted log records to subclass implementation. Subclass is free to either apply merging or expose raw data back to the caller. NOTE: If readBlockLazily is turned on, does not merge, instead keeps reading log blocks and merges everything at once This is an optimization to avoid seek() back and forth to read new block (forward seek()) and lazily read content of seen block (reverse and forward seek()) during merge | | Read Block 1 Metadata | | Read Block 1 Data | | | Read Block 2 Metadata | | Read Block 2 Data | | I/O Pass 1 | ..................... | I/O Pass 2 | ................. | | | Read Block N Metadata | | Read Block N Data |

This results in two I/O passes over the log file.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	AbstractHoodieLogRecordReader.Builder Builder used to build AbstractHoodieLogRecordScanner.
protected static interface	AbstractHoodieLogRecordReader.KeySpec Key specification with a list of column names.

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	forceFullScan
protected HoodieTableMetaClient	hoodieTableMetaClient
protected List<String>	logFilePaths
protected String	preCombineField
protected org.apache.avro.Schema	readerSchema
protected HoodieRecordMerger	recordMerger
protected HoodieRecord.HoodieRecordType	recordType

Constructor Summary

Constructors

Modifier

Constructor and Description

protected

AbstractHoodieLogRecordReader(HoodieStorage storage, String basePath, List<String> logFilePaths, org.apache.avro.Schema readerSchema, String latestInstantTime, boolean reverseReader, int bufferSize, Option<InstantRange> instantRange, boolean withOperationField, boolean forceFullScan, Option<String> partitionNameOverride, InternalSchema internalSchema, Option<String> keyFieldOverride, boolean enableOptimizedLogBlocksScan, HoodieRecordMerger recordMerger, Option<HoodieTableMetaClient> hoodieTableMetaClientOption)

Method Summary

Modifier and Type	Method and Description
Deque<HoodieLogBlock>	getCurrentInstantLogBlocks()
Option<String>	getPartitionNameOverride()
protected String	getPayloadClassFQN()
protected TypedProperties	getPayloadProps()
float	getProgress() Return progress of scanning as a float between 0.0 to 1.0.
long	getTotalCorruptBlocks()
long	getTotalLogBlocks()
long	getTotalLogFiles()
long	getTotalLogRecords()
long	getTotalRollbacks()
List<String>	getValidBlockInstants()
boolean	isWithOperationField()
protected abstract void	processNextDeletedRecord(DeleteRecord deleteRecord) Process next deleted record.
abstract <T> void	processNextRecord(HoodieRecord<T> hoodieRecord) Process next record.
protected void	scanInternal(Option<AbstractHoodieLogRecordReader.KeySpec> keySpecOpt, boolean skipProcessingBlocks)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

readerSchema

protected final org.apache.avro.Schema readerSchema

hoodieTableMetaClient

protected final HoodieTableMetaClient hoodieTableMetaClient

preCombineField

protected final String preCombineField

recordMerger

protected final HoodieRecordMerger recordMerger

logFilePaths

protected final List<String> logFilePaths

forceFullScan

protected final boolean forceFullScan

recordType

protected final HoodieRecord.HoodieRecordType recordType

Constructor Detail

AbstractHoodieLogRecordReader

protected AbstractHoodieLogRecordReader(HoodieStorage storage,
                                        String basePath,
                                        List<String> logFilePaths,
                                        org.apache.avro.Schema readerSchema,
                                        String latestInstantTime,
                                        boolean reverseReader,
                                        int bufferSize,
                                        Option<InstantRange> instantRange,
                                        boolean withOperationField,
                                        boolean forceFullScan,
                                        Option<String> partitionNameOverride,
                                        InternalSchema internalSchema,
                                        Option<String> keyFieldOverride,
                                        boolean enableOptimizedLogBlocksScan,
                                        HoodieRecordMerger recordMerger,
                                        Option<HoodieTableMetaClient> hoodieTableMetaClientOption)

Method Detail

scanInternal

protected final void scanInternal(Option<AbstractHoodieLogRecordReader.KeySpec> keySpecOpt,
                                  boolean skipProcessingBlocks)

Parameters:

keySpecOpt - specifies target set of keys to be scanned

skipProcessingBlocks - controls, whether (delta) blocks have to actually be processed

processNextRecord

public abstract <T> void processNextRecord(HoodieRecord<T> hoodieRecord)
                                    throws Exception

Process next record.

Parameters:

hoodieRecord - Hoodie Record to process

Throws:

Exception

processNextDeletedRecord

protected abstract void processNextDeletedRecord(DeleteRecord deleteRecord)

Process next deleted record.

Parameters:

deleteRecord - Deleted record(hoodie key and ordering value)

getProgress

public float getProgress()

Return progress of scanning as a float between 0.0 to 1.0.

getTotalLogFiles

public long getTotalLogFiles()

getTotalLogRecords

public long getTotalLogRecords()

getTotalLogBlocks

public long getTotalLogBlocks()

getPayloadClassFQN

protected String getPayloadClassFQN()

getPartitionNameOverride

public Option<String> getPartitionNameOverride()

getTotalRollbacks

public long getTotalRollbacks()

getTotalCorruptBlocks

public long getTotalCorruptBlocks()

isWithOperationField

public boolean isWithOperationField()

getPayloadProps

protected TypedProperties getPayloadProps()

getCurrentInstantLogBlocks

public Deque<HoodieLogBlock> getCurrentInstantLogBlocks()

getValidBlockInstants

public List<String> getValidBlockInstants()