org.apache.hudi.metadata

Class HoodieTableMetadataUtil

java.lang.Object

org.apache.hudi.metadata.HoodieTableMetadataUtil

public class HoodieTableMetadataUtil
extends Object

A utility to convert timeline information to metadata table records.

Field Summary

Fields

Modifier and Type	Field and Description
static String	PARTITION_NAME_BLOOM_FILTERS
static String	PARTITION_NAME_COLUMN_STATS
static String	PARTITION_NAME_FILES
static String	PARTITION_NAME_RECORD_INDEX

Method Summary

Modifier and Type	Method and Description
static Map<String,HoodieColumnRangeMetadata<Comparable>>	collectColumnRangeMetadata(List<org.apache.avro.generic.IndexedRecord> records, List<org.apache.avro.Schema.Field> targetFields, String filePath) Collects HoodieColumnRangeMetadata for the provided collection of records, pretending as if provided records have been persisted w/in given filePath
static HoodieColumnRangeMetadata<Comparable>	convertColumnStatsRecordToColumnRangeMetadata(HoodieMetadataColumnStats columnStats) Converts instance of HoodieMetadataColumnStats to HoodieColumnRangeMetadata
static HoodieData<HoodieRecord>	convertFilesToBloomFilterRecords(HoodieEngineContext engineContext, Map<String,List<String>> partitionToDeletedFiles, Map<String,Map<String,Long>> partitionToAppendedFiles, String instantTime, HoodieTableMetaClient dataMetaClient, int bloomIndexParallelism, String bloomFilterType) Convert added and deleted files metadata to bloom filter index records.
static HoodieData<HoodieRecord>	convertFilesToColumnStatsRecords(HoodieEngineContext engineContext, Map<String,List<String>> partitionToDeletedFiles, Map<String,Map<String,Long>> partitionToAppendedFiles, HoodieTableMetaClient dataMetaClient, boolean isColumnStatsIndexEnabled, int columnStatsIndexParallelism, List<String> targetColumnsForColumnStatsIndex) Convert added and deleted action metadata to column stats index records.
protected static List<HoodieRecord>	convertFilesToFilesPartitionRecords(Map<String,List<String>> partitionToDeletedFiles, Map<String,Map<String,Long>> partitionToAppendedFiles, String instantTime, String operation) Convert rollback action metadata to files partition records.
static HoodieData<HoodieRecord>	convertMetadataToBloomFilterRecords(HoodieCleanMetadata cleanMetadata, HoodieEngineContext engineContext, String instantTime, int bloomIndexParallelism) Convert clean metadata to bloom filter index records.
static HoodieData<HoodieRecord>	convertMetadataToBloomFilterRecords(HoodieEngineContext context, HoodieConfig hoodieConfig, HoodieCommitMetadata commitMetadata, String instantTime, HoodieTableMetaClient dataMetaClient, String bloomFilterType, int bloomIndexParallelism) Convert commit action metadata to bloom filter records.
static HoodieData<HoodieRecord>	convertMetadataToColumnStatsRecords(HoodieCleanMetadata cleanMetadata, HoodieEngineContext engineContext, HoodieTableMetaClient dataMetaClient, boolean isColumnStatsIndexEnabled, int columnStatsIndexParallelism, List<String> targetColumnsForColumnStatsIndex) Convert clean metadata to column stats index records.
static HoodieData<HoodieRecord>	convertMetadataToColumnStatsRecords(HoodieCommitMetadata commitMetadata, HoodieEngineContext engineContext, HoodieTableMetaClient dataMetaClient, boolean isColumnStatsIndexEnabled, int columnStatsIndexParallelism, List<String> targetColumnsForColumnStatsIndex)
static List<HoodieRecord>	convertMetadataToFilesPartitionRecords(HoodieCleanMetadata cleanMetadata, String instantTime) Finds all files that were deleted as part of a clean and creates metadata table records for them.
static List<HoodieRecord>	convertMetadataToFilesPartitionRecords(HoodieCommitMetadata commitMetadata, String instantTime) Finds all new files/partitions created as part of commit and creates metadata table records for them.
static Map<MetadataPartitionType,HoodieData<HoodieRecord>>	convertMetadataToRecords(HoodieEngineContext engineContext, HoodieCleanMetadata cleanMetadata, String instantTime, HoodieTableMetaClient dataMetaClient, List<MetadataPartitionType> enabledPartitionTypes, int bloomIndexParallelism, boolean isColumnStatsIndexEnabled, int columnStatsIndexParallelism, List<String> targetColumnsForColumnStatsIndex) Convert the clean action to metadata records.
static Map<MetadataPartitionType,HoodieData<HoodieRecord>>	convertMetadataToRecords(HoodieEngineContext context, HoodieConfig hoodieConfig, HoodieCommitMetadata commitMetadata, String instantTime, HoodieTableMetaClient dataMetaClient, List<MetadataPartitionType> enabledPartitionTypes, String bloomFilterType, int bloomIndexParallelism, boolean isColumnStatsIndexEnabled, int columnStatsIndexParallelism, List<String> targetColumnsForColumnStatsIndex) Convert commit action to metadata records for the enabled partition types.
static Map<MetadataPartitionType,HoodieData<HoodieRecord>>	convertMetadataToRecords(HoodieEngineContext engineContext, HoodieTableMetaClient dataTableMetaClient, HoodieRollbackMetadata rollbackMetadata, String instantTime) Convert rollback action metadata to metadata table records.
static Map<MetadataPartitionType,HoodieData<HoodieRecord>>	convertMissingPartitionRecords(HoodieEngineContext engineContext, List<String> deletedPartitions, Map<String,Map<String,Long>> filesAdded, Map<String,List<String>> filesDeleted, String instantTime)
static String	createAsyncIndexerTimestamp(String timestamp)
static String	createCleanTimestamp(String timestamp) Create the timestamp for a clean operation on the metadata table.
static String	createCompactionTimestamp(String timestamp) Create the timestamp for a compaction operation on the metadata table.
static String	createIndexInitTimestamp(String timestamp, int offset) Create the timestamp for an index initialization operation on the metadata table.
static String	createLogCompactionTimestamp(String timestamp) Create the timestamp for a compaction operation on the metadata table.
static String	createRestoreTimestamp(String timestamp)
static String	createRollbackTimestamp(String timestamp)
static void	deleteMetadataPartition(String basePath, HoodieEngineContext context, MetadataPartitionType partitionType) Deletes the metadata partition from the file system.
static String	deleteMetadataTable(HoodieTableMetaClient dataMetaClient, HoodieEngineContext context, boolean backup) Delete the metadata table for the dataset and backup if required.
static void	deleteMetadataTable(String basePath, HoodieEngineContext context) Delete the metadata table for the dataset.
static String	deleteMetadataTablePartition(HoodieTableMetaClient dataMetaClient, HoodieEngineContext context, MetadataPartitionType partitionType, boolean backup) Delete a partition within the metadata table.
static int	estimateFileGroupCount(MetadataPartitionType partitionType, long recordCount, int averageRecordSize, int minFileGroupCount, int maxFileGroupCount, float growthFactor, int maxFileGroupSizeBytes) Estimates the file group count to use for a MDT partition.
static String	getBloomFilterIndexPartitionIdentifier(String partitionName)
static String	getColumnStatsIndexPartitionIdentifier(String partitionName)
static int	getFileGroupIndexFromFileId(String fileId) Extract the index from the fileID of a file group in the MDT partition.
static String	getFileGroupPrefix(String fileId) Extract the fileID prefix from the fileID of a file group in the MDT partition.
static String	getFileIDForFileGroup(MetadataPartitionType partitionType, int index) Return the complete fileID for a file group within a MDT partition.
static HoodieTableFileSystemView	getFileSystemView(HoodieTableMetaClient metaClient) Get metadata table file system view.
static Set<String>	getInflightAndCompletedMetadataPartitions(HoodieTableConfig tableConfig)
static Set<String>	getInflightMetadataPartitions(HoodieTableConfig tableConfig)
static HoodieRecordGlobalLocation	getLocationFromRecordIndexInfo(HoodieRecordIndexInfo recordIndexInfo) Gets the location from record index content.
static HoodieRecordGlobalLocation	getLocationFromRecordIndexInfo(String partition, int fileIdEncoding, long fileIdHighBits, long fileIdLowBits, int fileIndex, String originalFileId, Long instantTime) Gets the location from record index content.
static boolean	getMetadataPartitionsNeedingWriteStatusTracking(HoodieMetadataConfig config, HoodieTableMetaClient metaClient) Returns true if any enabled metadata partition in the given hoodie table requires WriteStatus to track the written records.
static String	getPartitionIdentifierForFilesPartition(String relativePartitionPath)
static List<FileSlice>	getPartitionLatestFileSlices(HoodieTableMetaClient metaClient, Option<HoodieTableFileSystemView> fsView, String partition) Get the latest file slices for a Metadata Table partition.
static List<FileSlice>	getPartitionLatestFileSlicesIncludingInflight(HoodieTableMetaClient metaClient, Option<HoodieTableFileSystemView> fileSystemView, String partition) Get the latest file slices for a given partition including the inflight ones.
static List<FileSlice>	getPartitionLatestMergedFileSlices(HoodieTableMetaClient metaClient, HoodieTableFileSystemView fsView, String partition) Get the latest file slices for a Metadata Table partition.
static Set<String>	getValidInstantTimestamps(HoodieTableMetaClient dataMetaClient, HoodieTableMetaClient metadataMetaClient)
static Set<String>	getWritePartitionPaths(List<HoodieCommitMetadata> metadataList) Returns all the incremental write partition paths as a set with the given commits metadata.
static boolean	isFilesPartitionAvailable(HoodieTableMetaClient metaClient) Returns whether the files partition of metadata table is ready for read.
static boolean	isIndexingCommit(String instantTime) Checks if a delta commit in metadata table is written by async indexer.
static boolean	isValidInstant(HoodieInstant instant) Checks if the Instant is a delta commit and has a valid suffix for operations on MDT.
static int	mapRecordKeyToFileGroupIndex(String recordKey, int numFileGroups) Map a record key to a file group in partition of interest.
static boolean	metadataPartitionExists(String basePath, HoodieEngineContext context, MetadataPartitionType partitionType) Check if the given metadata partition exists.
static HoodieData<HoodieRecord>	readRecordKeysFromBaseFiles(HoodieEngineContext engineContext, HoodieConfig config, List<Pair<String,HoodieBaseFile>> partitionBaseFilePairs, boolean forDelete, int recordIndexMaxParallelism, String basePath, StorageConfiguration<?> configuration, String activeModule) Deprecated.
static HoodieData<HoodieRecord>	readRecordKeysFromFileSlices(HoodieEngineContext engineContext, List<Pair<String,FileSlice>> partitionFileSlicePairs, boolean forDelete, int recordIndexMaxParallelism, String activeModule, HoodieTableMetaClient metaClient, EngineType engineType) Reads the record keys from the given file slices and returns a HoodieData of HoodieRecord to be updated in the metadata table.
static BigDecimal	tryUpcastDecimal(BigDecimal value, org.apache.avro.LogicalTypes.Decimal decimal) Does an upcast for BigDecimal instance to align it with scale/precision expected by the LogicalTypes.Decimal Avro logical type

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PARTITION_NAME_FILES

public static final String PARTITION_NAME_FILES

See Also:

Constant Field Values

PARTITION_NAME_COLUMN_STATS

public static final String PARTITION_NAME_COLUMN_STATS

See Also:

Constant Field Values

PARTITION_NAME_BLOOM_FILTERS

public static final String PARTITION_NAME_BLOOM_FILTERS

See Also:

Constant Field Values

PARTITION_NAME_RECORD_INDEX

public static final String PARTITION_NAME_RECORD_INDEX

See Also:

Constant Field Values

Method Detail

isFilesPartitionAvailable

public static boolean isFilesPartitionAvailable(HoodieTableMetaClient metaClient)

Returns whether the files partition of metadata table is ready for read.

Parameters:

metaClient - HoodieTableMetaClient instance.

Returns:

true if the files partition of metadata table is ready for read, based on the table config; false otherwise.

collectColumnRangeMetadata

public static Map<String,HoodieColumnRangeMetadata<Comparable>> collectColumnRangeMetadata(List<org.apache.avro.generic.IndexedRecord> records,
                                                                                           List<org.apache.avro.Schema.Field> targetFields,
                                                                                           String filePath)

Collects HoodieColumnRangeMetadata for the provided collection of records, pretending as if provided records have been persisted w/in given filePath

Parameters:

records - target records to compute column range metadata for

targetFields - columns (fields) to be collected

filePath - file path value required for HoodieColumnRangeMetadata

Returns:

map of HoodieColumnRangeMetadata for each of the provided target fields for the collection of provided records

convertColumnStatsRecordToColumnRangeMetadata

public static HoodieColumnRangeMetadata<Comparable> convertColumnStatsRecordToColumnRangeMetadata(HoodieMetadataColumnStats columnStats)

Converts instance of HoodieMetadataColumnStats to HoodieColumnRangeMetadata

deleteMetadataTable

public static void deleteMetadataTable(String basePath,
                                       HoodieEngineContext context)

Delete the metadata table for the dataset. This will be invoked during upgrade/downgrade operation during which no other process should be running.

Parameters:

basePath - base path of the dataset

context - instance of HoodieEngineContext.

deleteMetadataPartition

public static void deleteMetadataPartition(String basePath,
                                           HoodieEngineContext context,
                                           MetadataPartitionType partitionType)

Deletes the metadata partition from the file system.

Parameters:

basePath - - base path of the dataset

context - - instance of HoodieEngineContext

partitionType - - MetadataPartitionType of the partition to delete

metadataPartitionExists

public static boolean metadataPartitionExists(String basePath,
                                              HoodieEngineContext context,
                                              MetadataPartitionType partitionType)

Check if the given metadata partition exists.

Parameters:

basePath - base path of the dataset

context - instance of HoodieEngineContext.

convertMetadataToRecords

public static Map<MetadataPartitionType,HoodieData<HoodieRecord>> convertMetadataToRecords(HoodieEngineContext context,
                                                                                           HoodieConfig hoodieConfig,
                                                                                           HoodieCommitMetadata commitMetadata,
                                                                                           String instantTime,
                                                                                           HoodieTableMetaClient dataMetaClient,
                                                                                           List<MetadataPartitionType> enabledPartitionTypes,
                                                                                           String bloomFilterType,
                                                                                           int bloomIndexParallelism,
                                                                                           boolean isColumnStatsIndexEnabled,
                                                                                           int columnStatsIndexParallelism,
                                                                                           List<String> targetColumnsForColumnStatsIndex)

Convert commit action to metadata records for the enabled partition types.

Parameters:

context - - Engine context to use

hoodieConfig - - Hudi configs

commitMetadata - - Commit action metadata

instantTime - - Action instant time

dataMetaClient - - HoodieTableMetaClient for data

enabledPartitionTypes - - List of enabled MDT partitions

bloomFilterType - - Type of generated bloom filter records

bloomIndexParallelism - - Parallelism for bloom filter record generation

isColumnStatsIndexEnabled - - Is column stats index enabled

columnStatsIndexParallelism - - Parallelism for column stats index records generation

targetColumnsForColumnStatsIndex - - List of columns for column stats index

Returns:

Map of partition to metadata records for the commit action

convertMetadataToFilesPartitionRecords

public static List<HoodieRecord> convertMetadataToFilesPartitionRecords(HoodieCommitMetadata commitMetadata,
                                                                        String instantTime)

Finds all new files/partitions created as part of commit and creates metadata table records for them.

Parameters:

commitMetadata - - Commit action metadata

instantTime - - Commit action instant time

Returns:

List of metadata table records

getWritePartitionPaths

public static Set<String> getWritePartitionPaths(List<HoodieCommitMetadata> metadataList)

Returns all the incremental write partition paths as a set with the given commits metadata.

Parameters:

metadataList - The commits metadata

Returns:

the partition path set

convertMetadataToBloomFilterRecords

public static HoodieData<HoodieRecord> convertMetadataToBloomFilterRecords(HoodieEngineContext context,
                                                                           HoodieConfig hoodieConfig,
                                                                           HoodieCommitMetadata commitMetadata,
                                                                           String instantTime,
                                                                           HoodieTableMetaClient dataMetaClient,
                                                                           String bloomFilterType,
                                                                           int bloomIndexParallelism)

Convert commit action metadata to bloom filter records.

Parameters:

context - - Engine context to use

hoodieConfig - - Hudi configs

commitMetadata - - Commit action metadata

instantTime - - Action instant time

dataMetaClient - - HoodieTableMetaClient for data

bloomFilterType - - Type of generated bloom filter records

bloomIndexParallelism - - Parallelism for bloom filter record generation

Returns:

HoodieData of metadata table records

convertMetadataToRecords

public static Map<MetadataPartitionType,HoodieData<HoodieRecord>> convertMetadataToRecords(HoodieEngineContext engineContext,
                                                                                           HoodieCleanMetadata cleanMetadata,
                                                                                           String instantTime,
                                                                                           HoodieTableMetaClient dataMetaClient,
                                                                                           List<MetadataPartitionType> enabledPartitionTypes,
                                                                                           int bloomIndexParallelism,
                                                                                           boolean isColumnStatsIndexEnabled,
                                                                                           int columnStatsIndexParallelism,
                                                                                           List<String> targetColumnsForColumnStatsIndex)

Convert the clean action to metadata records.

convertMetadataToFilesPartitionRecords

public static List<HoodieRecord> convertMetadataToFilesPartitionRecords(HoodieCleanMetadata cleanMetadata,
                                                                        String instantTime)

Finds all files that were deleted as part of a clean and creates metadata table records for them.

Parameters:

cleanMetadata -

instantTime -

Returns:

a list of metadata table records

convertMissingPartitionRecords

public static Map<MetadataPartitionType,HoodieData<HoodieRecord>> convertMissingPartitionRecords(HoodieEngineContext engineContext,
                                                                                                 List<String> deletedPartitions,
                                                                                                 Map<String,Map<String,Long>> filesAdded,
                                                                                                 Map<String,List<String>> filesDeleted,
                                                                                                 String instantTime)

convertMetadataToBloomFilterRecords

public static HoodieData<HoodieRecord> convertMetadataToBloomFilterRecords(HoodieCleanMetadata cleanMetadata,
                                                                           HoodieEngineContext engineContext,
                                                                           String instantTime,
                                                                           int bloomIndexParallelism)

Convert clean metadata to bloom filter index records.

Parameters:

cleanMetadata - - Clean action metadata

engineContext - - Engine context

instantTime - - Clean action instant time

bloomIndexParallelism - - Parallelism for bloom filter record generation

Returns:

List of bloom filter index records for the clean metadata

convertMetadataToColumnStatsRecords

public static HoodieData<HoodieRecord> convertMetadataToColumnStatsRecords(HoodieCleanMetadata cleanMetadata,
                                                                           HoodieEngineContext engineContext,
                                                                           HoodieTableMetaClient dataMetaClient,
                                                                           boolean isColumnStatsIndexEnabled,
                                                                           int columnStatsIndexParallelism,
                                                                           List<String> targetColumnsForColumnStatsIndex)

Convert clean metadata to column stats index records.

Parameters:

cleanMetadata - - Clean action metadata

engineContext - - Engine context

dataMetaClient - - HoodieTableMetaClient for data

isColumnStatsIndexEnabled - - Is column stats index enabled

columnStatsIndexParallelism - - Parallelism for column stats index records generation

targetColumnsForColumnStatsIndex - - List of columns for column stats index

Returns:

List of column stats index records for the clean metadata

convertMetadataToRecords

public static Map<MetadataPartitionType,HoodieData<HoodieRecord>> convertMetadataToRecords(HoodieEngineContext engineContext,
                                                                                           HoodieTableMetaClient dataTableMetaClient,
                                                                                           HoodieRollbackMetadata rollbackMetadata,
                                                                                           String instantTime)

Convert rollback action metadata to metadata table records.

We only need to handle FILES partition here as HUDI rollbacks on MOR table may end up adding a new log file. All other partitions are handled by actual rollback of the deltacommit which added records to those partitions.

convertFilesToFilesPartitionRecords

protected static List<HoodieRecord> convertFilesToFilesPartitionRecords(Map<String,List<String>> partitionToDeletedFiles,
                                                                        Map<String,Map<String,Long>> partitionToAppendedFiles,
                                                                        String instantTime,
                                                                        String operation)

Convert rollback action metadata to files partition records.

getColumnStatsIndexPartitionIdentifier

public static String getColumnStatsIndexPartitionIdentifier(String partitionName)

getBloomFilterIndexPartitionIdentifier

public static String getBloomFilterIndexPartitionIdentifier(String partitionName)

getPartitionIdentifierForFilesPartition

public static String getPartitionIdentifierForFilesPartition(String relativePartitionPath)

convertFilesToBloomFilterRecords

public static HoodieData<HoodieRecord> convertFilesToBloomFilterRecords(HoodieEngineContext engineContext,
                                                                        Map<String,List<String>> partitionToDeletedFiles,
                                                                        Map<String,Map<String,Long>> partitionToAppendedFiles,
                                                                        String instantTime,
                                                                        HoodieTableMetaClient dataMetaClient,
                                                                        int bloomIndexParallelism,
                                                                        String bloomFilterType)

Convert added and deleted files metadata to bloom filter index records.

convertFilesToColumnStatsRecords

public static HoodieData<HoodieRecord> convertFilesToColumnStatsRecords(HoodieEngineContext engineContext,
                                                                        Map<String,List<String>> partitionToDeletedFiles,
                                                                        Map<String,Map<String,Long>> partitionToAppendedFiles,
                                                                        HoodieTableMetaClient dataMetaClient,
                                                                        boolean isColumnStatsIndexEnabled,
                                                                        int columnStatsIndexParallelism,
                                                                        List<String> targetColumnsForColumnStatsIndex)

Convert added and deleted action metadata to column stats index records.

mapRecordKeyToFileGroupIndex

public static int mapRecordKeyToFileGroupIndex(String recordKey,
                                               int numFileGroups)

Map a record key to a file group in partition of interest.

Note: For hashing, the algorithm is same as String.hashCode() but is being defined here as hashCode() implementation is not guaranteed by the JVM to be consistent across JVM versions and implementations.

Parameters:

recordKey - record key for which the file group index is looked up for.

Returns:

An integer hash of the given string

getPartitionLatestMergedFileSlices

public static List<FileSlice> getPartitionLatestMergedFileSlices(HoodieTableMetaClient metaClient,
                                                                 HoodieTableFileSystemView fsView,
                                                                 String partition)

Get the latest file slices for a Metadata Table partition. If the file slice is because of pending compaction instant, then merge the file slice with the one just before the compaction instant time. The list of file slices returned is sorted in the correct order of file group name.

Parameters:

metaClient - Instance of HoodieTableMetaClient.

fsView - Metadata table filesystem view.

partition - The name of the partition whose file groups are to be loaded.

Returns:

List of latest file slices for all file groups in a given partition.

getPartitionLatestFileSlices

public static List<FileSlice> getPartitionLatestFileSlices(HoodieTableMetaClient metaClient,
                                                           Option<HoodieTableFileSystemView> fsView,
                                                           String partition)

Get the latest file slices for a Metadata Table partition. The list of file slices returned is sorted in the correct order of file group name.

Parameters:

metaClient - - Instance of HoodieTableMetaClient.

fsView - - Metadata table filesystem view

partition - - The name of the partition whose file groups are to be loaded.

Returns:

List of latest file slices for all file groups in a given partition.

getFileSystemView

public static HoodieTableFileSystemView getFileSystemView(HoodieTableMetaClient metaClient)

Get metadata table file system view.

Parameters:

metaClient - - Metadata table meta client

Returns:

Filesystem view for the metadata table

getPartitionLatestFileSlicesIncludingInflight

public static List<FileSlice> getPartitionLatestFileSlicesIncludingInflight(HoodieTableMetaClient metaClient,
                                                                            Option<HoodieTableFileSystemView> fileSystemView,
                                                                            String partition)

Get the latest file slices for a given partition including the inflight ones.

Parameters:

metaClient - - instance of HoodieTableMetaClient

fileSystemView - - hoodie table file system view, which will be fetched from meta client if not already present

partition - - name of the partition whose file groups are to be loaded

Returns:

convertMetadataToColumnStatsRecords

public static HoodieData<HoodieRecord> convertMetadataToColumnStatsRecords(HoodieCommitMetadata commitMetadata,
                                                                           HoodieEngineContext engineContext,
                                                                           HoodieTableMetaClient dataMetaClient,
                                                                           boolean isColumnStatsIndexEnabled,
                                                                           int columnStatsIndexParallelism,
                                                                           List<String> targetColumnsForColumnStatsIndex)

tryUpcastDecimal

public static BigDecimal tryUpcastDecimal(BigDecimal value,
                                          org.apache.avro.LogicalTypes.Decimal decimal)

Does an upcast for BigDecimal instance to align it with scale/precision expected by the LogicalTypes.Decimal Avro logical type

getInflightMetadataPartitions

public static Set<String> getInflightMetadataPartitions(HoodieTableConfig tableConfig)

getInflightAndCompletedMetadataPartitions

public static Set<String> getInflightAndCompletedMetadataPartitions(HoodieTableConfig tableConfig)

getValidInstantTimestamps

public static Set<String> getValidInstantTimestamps(HoodieTableMetaClient dataMetaClient,
                                                    HoodieTableMetaClient metadataMetaClient)

isValidInstant

public static boolean isValidInstant(HoodieInstant instant)

Checks if the Instant is a delta commit and has a valid suffix for operations on MDT.

Parameters:

instant - HoodieInstant to check.

Returns:

true if the instant is valid.

isIndexingCommit

public static boolean isIndexingCommit(String instantTime)

Checks if a delta commit in metadata table is written by async indexer.

TODO(HUDI-5733): This should be cleaned up once the proper fix of rollbacks in the metadata table is landed.

Parameters:

instantTime - Instant time to check.

Returns:

true if from async indexer; false otherwise.

deleteMetadataTable

public static String deleteMetadataTable(HoodieTableMetaClient dataMetaClient,
                                         HoodieEngineContext context,
                                         boolean backup)

Delete the metadata table for the dataset and backup if required.

Parameters:

dataMetaClient - HoodieTableMetaClient of the dataset for which metadata table is to be deleted

context - instance of HoodieEngineContext.

backup - Whether metadata table should be backed up before deletion. If true, the table is backed up to the directory with name metadata_.

Returns:

The backup directory if backup was requested

deleteMetadataTablePartition

public static String deleteMetadataTablePartition(HoodieTableMetaClient dataMetaClient,
                                                  HoodieEngineContext context,
                                                  MetadataPartitionType partitionType,
                                                  boolean backup)

Delete a partition within the metadata table.

This can be used to delete a partition so that it can be re-bootstrapped.

Parameters:

dataMetaClient - HoodieTableMetaClient of the dataset for which metadata table is to be deleted

context - instance of HoodieEngineContext.

backup - Whether metadata table should be backed up before deletion. If true, the table is backed up to the directory with name metadata_.

partitionType - The partition to delete

Returns:

The backup directory if backup was requested, null otherwise

getFileIDForFileGroup

public static String getFileIDForFileGroup(MetadataPartitionType partitionType,
                                           int index)

Return the complete fileID for a file group within a MDT partition.

MDT fileGroups have the format -. The fileIDPrefix is hardcoded for each MDT partition and index is an integer.

Parameters:

partitionType - The type of the MDT partition

index - Index of the file group within the partition

Returns:

The fileID

getFileGroupIndexFromFileId

public static int getFileGroupIndexFromFileId(String fileId)

Extract the index from the fileID of a file group in the MDT partition. See getFileIDForFileGroup for the format of the fileID.

Parameters:

fileId - fileID of a file group.

Returns:

The index of file group

getFileGroupPrefix

public static String getFileGroupPrefix(String fileId)

Extract the fileID prefix from the fileID of a file group in the MDT partition. See getFileIDForFileGroup for the format of the fileID.

Parameters:

fileId - fileID of a file group.

Returns:

The fileID without the file index

createCleanTimestamp

public static String createCleanTimestamp(String timestamp)

Create the timestamp for a clean operation on the metadata table.

createRollbackTimestamp

public static String createRollbackTimestamp(String timestamp)

createRestoreTimestamp

public static String createRestoreTimestamp(String timestamp)

createAsyncIndexerTimestamp

public static String createAsyncIndexerTimestamp(String timestamp)

createCompactionTimestamp

public static String createCompactionTimestamp(String timestamp)

Create the timestamp for a compaction operation on the metadata table.

createIndexInitTimestamp

public static String createIndexInitTimestamp(String timestamp,
                                              int offset)

Create the timestamp for an index initialization operation on the metadata table.

Since many MDT partitions can be initialized one after other the offset parameter controls generating a unique timestamp.

createLogCompactionTimestamp

public static String createLogCompactionTimestamp(String timestamp)

Create the timestamp for a compaction operation on the metadata table.

estimateFileGroupCount

public static int estimateFileGroupCount(MetadataPartitionType partitionType,
                                         long recordCount,
                                         int averageRecordSize,
                                         int minFileGroupCount,
                                         int maxFileGroupCount,
                                         float growthFactor,
                                         int maxFileGroupSizeBytes)

Estimates the file group count to use for a MDT partition.

Parameters:

partitionType - Type of the partition for which the file group count is to be estimated.

recordCount - The number of records expected to be written.

averageRecordSize - Average size of each record to be written.

minFileGroupCount - Minimum number of file groups to use.

maxFileGroupCount - Maximum number of file groups to use.

growthFactor - By what factor are the records (recordCount) expected to grow?

maxFileGroupSizeBytes - Maximum size of the file group.

Returns:

The estimated number of file groups.

getMetadataPartitionsNeedingWriteStatusTracking

public static boolean getMetadataPartitionsNeedingWriteStatusTracking(HoodieMetadataConfig config,
                                                                      HoodieTableMetaClient metaClient)

Returns true if any enabled metadata partition in the given hoodie table requires WriteStatus to track the written records.

Parameters:

config - MDT config

metaClient - HoodieTableMetaClient of the data table

Returns:

true if WriteStatus should track the written records else false.

getLocationFromRecordIndexInfo

public static HoodieRecordGlobalLocation getLocationFromRecordIndexInfo(HoodieRecordIndexInfo recordIndexInfo)

Gets the location from record index content.

Parameters:

recordIndexInfo - HoodieRecordIndexInfo instance.

Returns:

HoodieRecordGlobalLocation containing the location.

getLocationFromRecordIndexInfo

public static HoodieRecordGlobalLocation getLocationFromRecordIndexInfo(String partition,
                                                                        int fileIdEncoding,
                                                                        long fileIdHighBits,
                                                                        long fileIdLowBits,
                                                                        int fileIndex,
                                                                        String originalFileId,
                                                                        Long instantTime)

Gets the location from record index content. Note that, a UUID based fileId is stored as 3 pieces in record index (fileIdHighBits, fileIdLowBits and fileIndex). FileID format is {UUID}-{fileIndex}. The arguments are consistent with what HoodieRecordIndexInfo contains.

Parameters:

partition - The partition name the record belongs to.

fileIdEncoding - FileId encoding. Possible values are 0 and 1. O represents UUID based fileID, and 1 represents raw string format of the fileId.

fileIdHighBits - High 64 bits if the fileId is based on UUID format.

fileIdLowBits - Low 64 bits if the fileId is based on UUID format.

fileIndex - Index representing file index which is used to re-construct UUID based fileID.

originalFileId - FileId of the location where record belongs to. When the encoding is 1, fileID is stored in raw string format.

instantTime - Epoch time in millisecond representing the commit time at which record was added.

Returns:

HoodieRecordGlobalLocation containing the location.

readRecordKeysFromBaseFiles

@Deprecated
public static HoodieData<HoodieRecord> readRecordKeysFromBaseFiles(HoodieEngineContext engineContext,
                                                                               HoodieConfig config,
                                                                               List<Pair<String,HoodieBaseFile>> partitionBaseFilePairs,
                                                                               boolean forDelete,
                                                                               int recordIndexMaxParallelism,
                                                                               String basePath,
                                                                               StorageConfiguration<?> configuration,
                                                                               String activeModule)

Deprecated.

Reads the record keys from the base files and returns a HoodieData of HoodieRecord to be updated in the metadata table. Use readRecordKeysFromFileSlices(HoodieEngineContext, List, boolean, int, String, HoodieTableMetaClient, EngineType) instead.

readRecordKeysFromFileSlices

public static HoodieData<HoodieRecord> readRecordKeysFromFileSlices(HoodieEngineContext engineContext,
                                                                    List<Pair<String,FileSlice>> partitionFileSlicePairs,
                                                                    boolean forDelete,
                                                                    int recordIndexMaxParallelism,
                                                                    String activeModule,
                                                                    HoodieTableMetaClient metaClient,
                                                                    EngineType engineType)

Reads the record keys from the given file slices and returns a HoodieData of HoodieRecord to be updated in the metadata table. If file slice does not have any base file, then iterates over the log files to get the record keys.