org.apache.hudi.index.simple

Class HoodieSimpleIndex

java.lang.Object

org.apache.hudi.index.HoodieIndex<Object,Object>

org.apache.hudi.index.simple.HoodieSimpleIndex

All Implemented Interfaces:

Serializable

Direct Known Subclasses:

HoodieGlobalSimpleIndex

public class HoodieSimpleIndex
extends HoodieIndex<Object,Object>

A simple index which reads interested fields(record key and partition path) from base files and joins with incoming records to find the tagged location.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hudi.index.HoodieIndex

HoodieIndex.BucketIndexEngineType, HoodieIndex.IndexType

Field Summary

Fields inherited from class org.apache.hudi.index.HoodieIndex

config

Constructor Summary

Constructors

Constructor and Description
HoodieSimpleIndex(HoodieWriteConfig config, Option<BaseKeyGenerator> keyGeneratorOpt)

Method Summary

Modifier and Type	Method and Description
boolean	canIndexLogFiles() This is used by storage to determine, if its safe to send inserts, straight to the log, i.e having a FileSlice, with no data file.
protected HoodiePairData<HoodieKey,HoodieRecordLocation>	fetchRecordLocations(HoodieEngineContext context, HoodieTable hoodieTable, int parallelism, List<Pair<String,HoodieBaseFile>> baseFiles)
protected HoodiePairData<HoodieKey,HoodieRecordLocation>	fetchRecordLocationsForAffectedPartitions(HoodieData<HoodieKey> hoodieKeys, HoodieEngineContext context, HoodieTable hoodieTable, int parallelism) Fetch record locations for passed in HoodieKeys.
boolean	isGlobal() An index is `global` if HoodieKey to fileID mapping, does not depend on the `partitionPath`.
boolean	isImplicitWithStorage() An index is "implicit" with respect to storage, if just writing new data to a file slice, updates the index as well.
boolean	rollbackCommit(String commitTime) Rollback the effects of the commit made at instantTime.
<R> HoodieData<HoodieRecord<R>>	tagLocation(HoodieData<HoodieRecord<R>> records, HoodieEngineContext context, HoodieTable hoodieTable) Looks up the index and tags each incoming record with a location of a file that contains the row (if it is actually present).
protected <R> HoodieData<HoodieRecord<R>>	tagLocationInternal(HoodieData<HoodieRecord<R>> inputRecords, HoodieEngineContext context, HoodieTable hoodieTable) Tags records location for incoming records.
HoodieData<WriteStatus>	updateLocation(HoodieData<WriteStatus> writeStatuses, HoodieEngineContext context, HoodieTable hoodieTable) Extracts the location of written records, and updates the index.

Methods inherited from class org.apache.hudi.index.HoodieIndex

close, requiresTagging, tagLocation, updateLocation

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HoodieSimpleIndex

public HoodieSimpleIndex(HoodieWriteConfig config,
                         Option<BaseKeyGenerator> keyGeneratorOpt)

Method Detail

updateLocation

public HoodieData<WriteStatus> updateLocation(HoodieData<WriteStatus> writeStatuses,
                                              HoodieEngineContext context,
                                              HoodieTable hoodieTable)

Description copied from class: HoodieIndex

Extracts the location of written records, and updates the index.

Specified by:

updateLocation in class HoodieIndex<Object,Object>

rollbackCommit

public boolean rollbackCommit(String commitTime)

Description copied from class: HoodieIndex

Rollback the effects of the commit made at instantTime.

Specified by:

rollbackCommit in class HoodieIndex<Object,Object>

isGlobal

public boolean isGlobal()

Description copied from class: HoodieIndex

An index is `global` if HoodieKey to fileID mapping, does not depend on the `partitionPath`. Such an implementation is able to obtain the same mapping, for two hoodie keys with same `recordKey` but different `partitionPath`

Specified by:

isGlobal in class HoodieIndex<Object,Object>

Returns:

whether or not, the index implementation is global in nature

canIndexLogFiles

public boolean canIndexLogFiles()

Description copied from class: HoodieIndex

This is used by storage to determine, if its safe to send inserts, straight to the log, i.e having a FileSlice, with no data file.

Specified by:

canIndexLogFiles in class HoodieIndex<Object,Object>

Returns:

Returns true/false depending on whether the impl has this capability

isImplicitWithStorage

public boolean isImplicitWithStorage()

Description copied from class: HoodieIndex

An index is "implicit" with respect to storage, if just writing new data to a file slice, updates the index as well. This is used by storage, to save memory footprint in certain cases.

Specified by:

isImplicitWithStorage in class HoodieIndex<Object,Object>

tagLocation

public <R> HoodieData<HoodieRecord<R>> tagLocation(HoodieData<HoodieRecord<R>> records,
                                                   HoodieEngineContext context,
                                                   HoodieTable hoodieTable)

Description copied from class: HoodieIndex

Looks up the index and tags each incoming record with a location of a file that contains the row (if it is actually present).

Specified by:

tagLocation in class HoodieIndex<Object,Object>

tagLocationInternal

protected <R> HoodieData<HoodieRecord<R>> tagLocationInternal(HoodieData<HoodieRecord<R>> inputRecords,
                                                              HoodieEngineContext context,
                                                              HoodieTable hoodieTable)

Tags records location for incoming records.

Parameters:

inputRecords - HoodieData of incoming records

context - instance of HoodieEngineContext to use

hoodieTable - instance of HoodieTable to use

Returns:

HoodieData of records with record locations set

fetchRecordLocationsForAffectedPartitions

protected HoodiePairData<HoodieKey,HoodieRecordLocation> fetchRecordLocationsForAffectedPartitions(HoodieData<HoodieKey> hoodieKeys,
                                                                                                   HoodieEngineContext context,
                                                                                                   HoodieTable hoodieTable,
                                                                                                   int parallelism)

Fetch record locations for passed in HoodieKeys.

Parameters:

hoodieKeys - HoodieData of HoodieKeys for which locations are fetched

context - instance of HoodieEngineContext to use

hoodieTable - instance of HoodieTable of interest

parallelism - parallelism to use

Returns:

HoodiePairData of HoodieKey and HoodieRecordLocation

fetchRecordLocations

protected HoodiePairData<HoodieKey,HoodieRecordLocation> fetchRecordLocations(HoodieEngineContext context,
                                                                              HoodieTable hoodieTable,
                                                                              int parallelism,
                                                                              List<Pair<String,HoodieBaseFile>> baseFiles)