org.apache.hudi.metadata

Class HoodieMetadataMergedLogRecordReader

java.lang.Object

org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader

org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner

org.apache.hudi.metadata.HoodieMetadataMergedLogRecordReader

All Implemented Interfaces:

Iterable<HoodieRecord<? extends HoodieRecordPayload>>

public class HoodieMetadataMergedLogRecordReader
extends HoodieMergedLogRecordScanner

A HoodieMergedLogRecordScanner implementation which only merged records matching providing keys. This is useful in limiting memory usage when only a small subset of updates records are to be read.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HoodieMetadataMergedLogRecordReader.Builder Builder used to build HoodieMetadataMergedLogRecordScanner.

Nested classes/interfaces inherited from class org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader

AbstractHoodieLogRecordReader.KeySpec

Field Summary

Fields inherited from class org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner

records, timer

Fields inherited from class org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader

forceFullScan, logFilePaths, readerSchema

Method Summary

Modifier and Type	Method and Description
protected HoodieAvroRecord<?>	createHoodieRecord(org.apache.avro.generic.IndexedRecord rec, HoodieTableConfig hoodieTableConfig, String payloadClassFQN, String preCombineField, boolean withOperationField, Option<Pair<String,String>> simpleKeyGenFields, Option<String> partitionName) Create @HoodieRecord from the @IndexedRecord.
protected String	getKeyField()
List<Pair<String,Option<HoodieRecord<HoodieMetadataPayload>>>>	getRecordByKey(String key) Retrieve a record given its key.
List<HoodieRecord<HoodieMetadataPayload>>	getRecordsByKeyPrefixes(List<String> keyPrefixes)
List<Pair<String,Option<HoodieRecord<HoodieMetadataPayload>>>>	getRecordsByKeys(List<String> keys)
static HoodieMetadataMergedLogRecordReader.Builder	newBuilder() Returns the builder for HoodieMetadataMergedLogRecordScanner.

Methods inherited from class org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner

close, getNumMergedRecordsInLog, getRecords, getTotalTimeTakenToReadAndMergeBlocks, iterator, performScan, processNextDeletedRecord, processNextRecord

Methods inherited from class org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader

getPartitionName, getPayloadClassFQN, getProgress, getTotalCorruptBlocks, getTotalLogBlocks, getTotalLogFiles, getTotalLogRecords, getTotalRollbacks, isWithOperationField, scan, scan, scanInternal

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Method Detail

createHoodieRecord

protected HoodieAvroRecord<?> createHoodieRecord(org.apache.avro.generic.IndexedRecord rec,
                                                 HoodieTableConfig hoodieTableConfig,
                                                 String payloadClassFQN,
                                                 String preCombineField,
                                                 boolean withOperationField,
                                                 Option<Pair<String,String>> simpleKeyGenFields,
                                                 Option<String> partitionName)

Description copied from class: AbstractHoodieLogRecordReader

Create @HoodieRecord from the @IndexedRecord.

Overrides:

createHoodieRecord in class AbstractHoodieLogRecordReader

Parameters:

rec - - IndexedRecord to create the HoodieRecord from

hoodieTableConfig - - Table config

payloadClassFQN - - Payload class fully qualified name

preCombineField - - PreCombine field

withOperationField - - Whether operation field is enabled

simpleKeyGenFields - - Key generator fields when populate meta fields is tuened off

partitionName - - Partition name

Returns:

HoodieRecord created from the IndexedRecord

newBuilder

public static HoodieMetadataMergedLogRecordReader.Builder newBuilder()

Returns the builder for HoodieMetadataMergedLogRecordScanner.

getRecordByKey

public List<Pair<String,Option<HoodieRecord<HoodieMetadataPayload>>>> getRecordByKey(String key)

Retrieve a record given its key.

Parameters:

key - Key of the record to retrieve

Returns:

HoodieRecord if key was found else Option.empty()

getRecordsByKeyPrefixes

public List<HoodieRecord<HoodieMetadataPayload>> getRecordsByKeyPrefixes(List<String> keyPrefixes)

getRecordsByKeys

public List<Pair<String,Option<HoodieRecord<HoodieMetadataPayload>>>> getRecordsByKeys(List<String> keys)

getKeyField

protected String getKeyField()

Overrides:

getKeyField in class AbstractHoodieLogRecordReader