org.apache.hudi.index

Class HoodieIndexUtils

java.lang.Object

org.apache.hudi.index.HoodieIndexUtils

public class HoodieIndexUtils
extends Object

Hoodie Index Utilities.

Constructor Summary

Constructors

Constructor and Description
HoodieIndexUtils()

Method Summary

Modifier and Type	Method and Description
static boolean	checkIfValidCommit(HoodieTimeline commitTimeline, String commitTs) Check if the given commit timestamp is valid for the timeline.
static <R> HoodieRecord<R>	createNewTaggedHoodieRecord(HoodieRecord<R> oldRecord, HoodieRecordGlobalLocation location, HoodieRecord.HoodieRecordType recordType)
static HoodieIndex	createUserDefinedIndex(HoodieWriteConfig config)
static List<String>	filterKeysFromFile(org.apache.hadoop.fs.Path filePath, List<String> candidateRecordKeys, org.apache.hadoop.conf.Configuration configuration) Given a list of row keys and one file, return only row keys existing in that file.
static List<Pair<String,HoodieBaseFile>>	getLatestBaseFilesForAllPartitions(List<String> partitions, HoodieEngineContext context, HoodieTable hoodieTable) Fetches Pair of partition path and HoodieBaseFiles for interested partitions.
static List<HoodieBaseFile>	getLatestBaseFilesForPartition(String partition, HoodieTable hoodieTable) Fetches Pair of partition path and HoodieBaseFiles for interested partitions.
static List<FileSlice>	getLatestFileSlicesForPartition(String partition, HoodieTable hoodieTable) Fetches Pair of partition path and FileSlices for interested partitions.
static <R> HoodieData<HoodieRecord<R>>	mergeForPartitionUpdatesIfNeeded(HoodieData<Pair<HoodieRecord<R>,Option<HoodieRecordGlobalLocation>>> incomingRecordsAndLocations, HoodieWriteConfig config, HoodieTable hoodieTable) Merge tagged incoming records with existing records in case of partition path updated.
static <R> HoodieRecord<R>	tagAsNewRecordIfNeeded(HoodieRecord<R> record, Option<HoodieRecordLocation> location) Get tagged record for the passed in HoodieRecord.
static <R> HoodieData<HoodieRecord<R>>	tagGlobalLocationBackToRecords(HoodieData<HoodieRecord<R>> incomingRecords, HoodiePairData<String,HoodieRecordGlobalLocation> keyAndExistingLocations, boolean mayContainDuplicateLookup, boolean shouldUpdatePartitionPath, HoodieWriteConfig config, HoodieTable table)
static <R> HoodieRecord<R>	tagRecord(HoodieRecord<R> record, HoodieRecordLocation location) Tag the record to an existing location.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HoodieIndexUtils

public HoodieIndexUtils()

Method Detail

getLatestBaseFilesForPartition

public static List<HoodieBaseFile> getLatestBaseFilesForPartition(String partition,
                                                                  HoodieTable hoodieTable)

Fetches Pair of partition path and HoodieBaseFiles for interested partitions.

Parameters:

partition - Partition of interest

hoodieTable - Instance of HoodieTable of interest

Returns:

the list of HoodieBaseFile

getLatestFileSlicesForPartition

public static List<FileSlice> getLatestFileSlicesForPartition(String partition,
                                                              HoodieTable hoodieTable)

Fetches Pair of partition path and FileSlices for interested partitions.

Parameters:

partition - Partition of interest

hoodieTable - Instance of HoodieTable of interest

Returns:

the list of FileSlice

getLatestBaseFilesForAllPartitions

public static List<Pair<String,HoodieBaseFile>> getLatestBaseFilesForAllPartitions(List<String> partitions,
                                                                                   HoodieEngineContext context,
                                                                                   HoodieTable hoodieTable)

Fetches Pair of partition path and HoodieBaseFiles for interested partitions.

Parameters:

partitions - list of partitions of interest

context - instance of HoodieEngineContext to use

hoodieTable - instance of HoodieTable of interest

Returns:

the list of Pairs of partition path and fileId

tagAsNewRecordIfNeeded

public static <R> HoodieRecord<R> tagAsNewRecordIfNeeded(HoodieRecord<R> record,
                                                         Option<HoodieRecordLocation> location)

Get tagged record for the passed in HoodieRecord.

Parameters:

record - instance of HoodieRecord for which tagging is requested

location - HoodieRecordLocation for the passed in HoodieRecord

Returns:

the tagged HoodieRecord

tagRecord

public static <R> HoodieRecord<R> tagRecord(HoodieRecord<R> record,
                                            HoodieRecordLocation location)

Tag the record to an existing location. Not creating any new instance.

filterKeysFromFile

public static List<String> filterKeysFromFile(org.apache.hadoop.fs.Path filePath,
                                              List<String> candidateRecordKeys,
                                              org.apache.hadoop.conf.Configuration configuration)
                                       throws HoodieIndexException

Given a list of row keys and one file, return only row keys existing in that file.

Parameters:

filePath - - File to filter keys from

candidateRecordKeys - - Candidate keys to filter

Returns:

List of candidate keys that are available in the file

Throws:

HoodieIndexException

checkIfValidCommit

public static boolean checkIfValidCommit(HoodieTimeline commitTimeline,
                                         String commitTs)

Check if the given commit timestamp is valid for the timeline. The commit timestamp is considered to be valid if: 1. the commit timestamp is present in the timeline, or 2. the commit timestamp is less than the first commit timestamp in the timeline

Parameters:

commitTimeline - The timeline

commitTs - The commit timestamp to check

Returns:

true if the commit timestamp is valid for the timeline

createUserDefinedIndex

public static HoodieIndex createUserDefinedIndex(HoodieWriteConfig config)

mergeForPartitionUpdatesIfNeeded

public static <R> HoodieData<HoodieRecord<R>> mergeForPartitionUpdatesIfNeeded(HoodieData<Pair<HoodieRecord<R>,Option<HoodieRecordGlobalLocation>>> incomingRecordsAndLocations,
                                                                               HoodieWriteConfig config,
                                                                               HoodieTable hoodieTable)

Merge tagged incoming records with existing records in case of partition path updated.

tagGlobalLocationBackToRecords

public static <R> HoodieData<HoodieRecord<R>> tagGlobalLocationBackToRecords(HoodieData<HoodieRecord<R>> incomingRecords,
                                                                             HoodiePairData<String,HoodieRecordGlobalLocation> keyAndExistingLocations,
                                                                             boolean mayContainDuplicateLookup,
                                                                             boolean shouldUpdatePartitionPath,
                                                                             HoodieWriteConfig config,
                                                                             HoodieTable table)

createNewTaggedHoodieRecord

public static <R> HoodieRecord<R> createNewTaggedHoodieRecord(HoodieRecord<R> oldRecord,
                                                              HoodieRecordGlobalLocation location,
                                                              HoodieRecord.HoodieRecordType recordType)