org.apache.hudi.index.bloom

Class HoodieBloomIndex

java.lang.Object

org.apache.hudi.index.HoodieIndex<Object,Object>

org.apache.hudi.index.bloom.HoodieBloomIndex

All Implemented Interfaces:

Serializable

Direct Known Subclasses:

HoodieGlobalBloomIndex

public class HoodieBloomIndex
extends HoodieIndex<Object,Object>

Indexing mechanism based on bloom filter. Each parquet file includes its row_key bloom filter in its metadata.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hudi.index.HoodieIndex

HoodieIndex.BucketIndexEngineType, HoodieIndex.IndexType

Field Summary

Fields inherited from class org.apache.hudi.index.HoodieIndex

config

Constructor Summary

Constructors

Constructor and Description
HoodieBloomIndex(HoodieWriteConfig config, BaseHoodieBloomIndexHelper bloomIndexHelper)

Method Summary

Modifier and Type	Method and Description
boolean	canIndexLogFiles() No indexes into log files yet.
boolean	isGlobal() This is not global, since we depend on the partitionPath to do the lookup.
boolean	isImplicitWithStorage() Bloom filters are stored, into the same data files.
protected List<Pair<String,BloomIndexFileInfo>>	loadColumnRangesFromMetaIndex(List<String> partitions, HoodieEngineContext context, HoodieTable<?,?,?,?> hoodieTable) Load the column stats index as BloomIndexFileInfo for all the involved files in the partition.
boolean	rollbackCommit(String instantTime) Rollback the effects of the commit made at instantTime.
<R> HoodieData<HoodieRecord<R>>	tagLocation(HoodieData<HoodieRecord<R>> records, HoodieEngineContext context, HoodieTable hoodieTable) Looks up the index and tags each incoming record with a location of a file that contains the row (if it is actually present).
protected <R> HoodieData<HoodieRecord<R>>	tagLocationBacktoRecords(HoodiePairData<HoodieKey,HoodieRecordLocation> keyFilenamePair, HoodieData<HoodieRecord<R>> records) Tag the back to the original HoodieRecord List.
HoodieData<WriteStatus>	updateLocation(HoodieData<WriteStatus> writeStatusData, HoodieEngineContext context, HoodieTable hoodieTable) Extracts the location of written records, and updates the index.

Methods inherited from class org.apache.hudi.index.HoodieIndex

close, requiresTagging, tagLocation, updateLocation

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HoodieBloomIndex

public HoodieBloomIndex(HoodieWriteConfig config,
                        BaseHoodieBloomIndexHelper bloomIndexHelper)

Method Detail

tagLocation

public <R> HoodieData<HoodieRecord<R>> tagLocation(HoodieData<HoodieRecord<R>> records,
                                                   HoodieEngineContext context,
                                                   HoodieTable hoodieTable)

Description copied from class: HoodieIndex

Looks up the index and tags each incoming record with a location of a file that contains the row (if it is actually present).

Specified by:

tagLocation in class HoodieIndex<Object,Object>

loadColumnRangesFromMetaIndex

protected List<Pair<String,BloomIndexFileInfo>> loadColumnRangesFromMetaIndex(List<String> partitions,
                                                                              HoodieEngineContext context,
                                                                              HoodieTable<?,?,?,?> hoodieTable)

Load the column stats index as BloomIndexFileInfo for all the involved files in the partition.

Parameters:

partitions - - List of partitions for which column stats need to be loaded

context - - Engine context

hoodieTable - - Hoodie table

Returns:

List of partition and file column range info pairs

rollbackCommit

public boolean rollbackCommit(String instantTime)

Description copied from class: HoodieIndex

Rollback the effects of the commit made at instantTime.

Specified by:

rollbackCommit in class HoodieIndex<Object,Object>

isGlobal

public boolean isGlobal()

This is not global, since we depend on the partitionPath to do the lookup.

Specified by:

isGlobal in class HoodieIndex<Object,Object>

Returns:

whether or not, the index implementation is global in nature

canIndexLogFiles

public boolean canIndexLogFiles()

No indexes into log files yet.

Specified by:

canIndexLogFiles in class HoodieIndex<Object,Object>

Returns:

Returns true/false depending on whether the impl has this capability

isImplicitWithStorage

public boolean isImplicitWithStorage()

Bloom filters are stored, into the same data files.

Specified by:

isImplicitWithStorage in class HoodieIndex<Object,Object>

tagLocationBacktoRecords

protected <R> HoodieData<HoodieRecord<R>> tagLocationBacktoRecords(HoodiePairData<HoodieKey,HoodieRecordLocation> keyFilenamePair,
                                                                   HoodieData<HoodieRecord<R>> records)

Tag the back to the original HoodieRecord List.

updateLocation

public HoodieData<WriteStatus> updateLocation(HoodieData<WriteStatus> writeStatusData,
                                              HoodieEngineContext context,
                                              HoodieTable hoodieTable)

Description copied from class: HoodieIndex

Extracts the location of written records, and updates the index.

Specified by:

updateLocation in class HoodieIndex<Object,Object>