org.apache.hudi.common.table.log

Class HoodieMergedLogRecordScanner

java.lang.Object

org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader

org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner

All Implemented Interfaces:

Iterable<HoodieRecord<? extends HoodieRecordPayload>>

Direct Known Subclasses:

HoodieMetadataMergedLogRecordReader

public class HoodieMergedLogRecordScanner
extends AbstractHoodieLogRecordReader
implements Iterable<HoodieRecord<? extends HoodieRecordPayload>>

Scans through all the blocks in a list of HoodieLogFile and builds up a compacted/merged list of records which will be used as a lookup table when merging the base columnar file with the redo log file.

NOTE: If readBlockLazily is turned on, does not merge, instead keeps reading log blocks and merges everything at once This is an optimization to avoid seek() back and forth to read new block (forward seek()) and lazily read content of seen block (reverse and forward seek()) during merge | | Read Block 1 Metadata | | Read Block 1 Data | | | Read Block 2 Metadata | | Read Block 2 Data | | I/O Pass 1 | ..................... | I/O Pass 2 | ................. | | | Read Block N Metadata | | Read Block N Data |

This results in two I/O passes over the log file.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HoodieMergedLogRecordScanner.Builder Builder used to build HoodieUnMergedLogRecordScanner.

Nested classes/interfaces inherited from class org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader

AbstractHoodieLogRecordReader.KeySpec

Field Summary

Fields

Modifier and Type	Field and Description
protected ExternalSpillableMap<String,HoodieRecord<? extends HoodieRecordPayload>>	records
HoodieTimer	timer

Fields inherited from class org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader

forceFullScan, logFilePaths, readerSchema

Constructor Summary

Constructors

Modifier

Constructor and Description

protected

HoodieMergedLogRecordScanner(org.apache.hadoop.fs.FileSystem fs, String basePath, List<String> logFilePaths, org.apache.avro.Schema readerSchema, String latestInstantTime, Long maxMemorySizeInBytes, boolean readBlocksLazily, boolean reverseReader, int bufferSize, String spillableMapBasePath, Option<InstantRange> instantRange, ExternalSpillableMap.DiskMapType diskMapType, boolean isBitCaskDiskMapCompressionEnabled, boolean withOperationField, boolean forceFullScan, Option<String> partitionName, InternalSchema internalSchema)

Method Summary

Modifier and Type	Method and Description
void	close()
long	getNumMergedRecordsInLog()
Map<String,HoodieRecord<? extends HoodieRecordPayload>>	getRecords()
long	getTotalTimeTakenToReadAndMergeBlocks()
Iterator<HoodieRecord<? extends HoodieRecordPayload>>	iterator()
static HoodieMergedLogRecordScanner.Builder	newBuilder() Returns the builder for HoodieMergedLogRecordScanner.
protected void	performScan()
protected void	processNextDeletedRecord(DeleteRecord deleteRecord) Process next deleted record.
protected void	processNextRecord(HoodieRecord<? extends HoodieRecordPayload> hoodieRecord) Process next record.

Methods inherited from class org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader

createHoodieRecord, getKeyField, getPartitionName, getPayloadClassFQN, getProgress, getTotalCorruptBlocks, getTotalLogBlocks, getTotalLogFiles, getTotalLogRecords, getTotalRollbacks, isWithOperationField, scan, scan, scanInternal

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Field Detail

timer

public final HoodieTimer timer

records

protected final ExternalSpillableMap<String,HoodieRecord<? extends HoodieRecordPayload>> records

Constructor Detail

HoodieMergedLogRecordScanner

protected HoodieMergedLogRecordScanner(org.apache.hadoop.fs.FileSystem fs,
                                       String basePath,
                                       List<String> logFilePaths,
                                       org.apache.avro.Schema readerSchema,
                                       String latestInstantTime,
                                       Long maxMemorySizeInBytes,
                                       boolean readBlocksLazily,
                                       boolean reverseReader,
                                       int bufferSize,
                                       String spillableMapBasePath,
                                       Option<InstantRange> instantRange,
                                       ExternalSpillableMap.DiskMapType diskMapType,
                                       boolean isBitCaskDiskMapCompressionEnabled,
                                       boolean withOperationField,
                                       boolean forceFullScan,
                                       Option<String> partitionName,
                                       InternalSchema internalSchema)

Method Detail

performScan

protected void performScan()

iterator

public Iterator<HoodieRecord<? extends HoodieRecordPayload>> iterator()

Specified by:

iterator in interface Iterable<HoodieRecord<? extends HoodieRecordPayload>>

getRecords

public Map<String,HoodieRecord<? extends HoodieRecordPayload>> getRecords()

getNumMergedRecordsInLog

public long getNumMergedRecordsInLog()

newBuilder

public static HoodieMergedLogRecordScanner.Builder newBuilder()

Returns the builder for HoodieMergedLogRecordScanner.

processNextRecord

protected void processNextRecord(HoodieRecord<? extends HoodieRecordPayload> hoodieRecord)
                          throws IOException

Description copied from class: AbstractHoodieLogRecordReader

Process next record.

Specified by:

processNextRecord in class AbstractHoodieLogRecordReader

Parameters:

hoodieRecord - Hoodie Record to process

Throws:

IOException

processNextDeletedRecord

protected void processNextDeletedRecord(DeleteRecord deleteRecord)

Description copied from class: AbstractHoodieLogRecordReader

Process next deleted record.

Specified by:

processNextDeletedRecord in class AbstractHoodieLogRecordReader

Parameters:

deleteRecord - Deleted record(hoodie key and ordering value)

getTotalTimeTakenToReadAndMergeBlocks

public long getTotalTimeTakenToReadAndMergeBlocks()

close

public void close()