org.apache.hudi.metadata

Class HoodieTableMetadataUtil

java.lang.Object

org.apache.hudi.metadata.HoodieTableMetadataUtil

public class HoodieTableMetadataUtil
extends Object

A utility to convert timeline information to metadata table records.

Field Summary

Fields

Modifier and Type	Field and Description
static String	PARTITION_NAME_BLOOM_FILTERS
static String	PARTITION_NAME_COLUMN_STATS
static String	PARTITION_NAME_FILES

Constructor Summary

Constructors

Constructor and Description
HoodieTableMetadataUtil()

Method Summary

Modifier and Type	Method and Description
static Map<String,HoodieColumnRangeMetadata<Comparable>>	collectColumnRangeMetadata(List<org.apache.avro.generic.IndexedRecord> records, List<org.apache.avro.Schema.Field> targetFields, String filePath) Collects HoodieColumnRangeMetadata for the provided collection of records, pretending as if provided records have been persisted w/in given filePath
static HoodieColumnRangeMetadata<Comparable>	convertColumnStatsRecordToColumnRangeMetadata(HoodieMetadataColumnStats columnStats) Converts instance of HoodieMetadataColumnStats to HoodieColumnRangeMetadata
static HoodieData<HoodieRecord>	convertFilesToBloomFilterRecords(HoodieEngineContext engineContext, Map<String,List<String>> partitionToDeletedFiles, Map<String,Map<String,Long>> partitionToAppendedFiles, MetadataRecordsGenerationParams recordsGenerationParams, String instantTime) Convert added and deleted files metadata to bloom filter index records.
static HoodieData<HoodieRecord>	convertFilesToColumnStatsRecords(HoodieEngineContext engineContext, Map<String,List<String>> partitionToDeletedFiles, Map<String,Map<String,Long>> partitionToAppendedFiles, MetadataRecordsGenerationParams recordsGenerationParams) Convert added and deleted action metadata to column stats index records.
static HoodieData<HoodieRecord>	convertMetadataToBloomFilterRecords(HoodieCleanMetadata cleanMetadata, HoodieEngineContext engineContext, String instantTime, MetadataRecordsGenerationParams recordsGenerationParams) Convert clean metadata to bloom filter index records.
static HoodieData<HoodieRecord>	convertMetadataToBloomFilterRecords(HoodieEngineContext context, HoodieCommitMetadata commitMetadata, String instantTime, MetadataRecordsGenerationParams recordsGenerationParams) Convert commit action metadata to bloom filter records.
static HoodieData<HoodieRecord>	convertMetadataToColumnStatsRecords(HoodieCleanMetadata cleanMetadata, HoodieEngineContext engineContext, MetadataRecordsGenerationParams recordsGenerationParams) Convert clean metadata to column stats index records.
static HoodieData<HoodieRecord>	convertMetadataToColumnStatsRecords(HoodieCommitMetadata commitMetadata, HoodieEngineContext engineContext, MetadataRecordsGenerationParams recordsGenerationParams)
static List<HoodieRecord>	convertMetadataToFilesPartitionRecords(HoodieCleanMetadata cleanMetadata, String instantTime) Finds all files that were deleted as part of a clean and creates metadata table records for them.
static List<HoodieRecord>	convertMetadataToFilesPartitionRecords(HoodieCommitMetadata commitMetadata, String instantTime) Finds all new files/partitions created as part of commit and creates metadata table records for them.
static Map<MetadataPartitionType,HoodieData<HoodieRecord>>	convertMetadataToRecords(HoodieEngineContext engineContext, HoodieActiveTimeline metadataTableTimeline, HoodieRestoreMetadata restoreMetadata, MetadataRecordsGenerationParams recordsGenerationParams, String instantTime, Option<String> lastSyncTs) Convert restore action metadata to metadata table records.
static Map<MetadataPartitionType,HoodieData<HoodieRecord>>	convertMetadataToRecords(HoodieEngineContext engineContext, HoodieActiveTimeline metadataTableTimeline, HoodieRollbackMetadata rollbackMetadata, MetadataRecordsGenerationParams recordsGenerationParams, String instantTime, Option<String> lastSyncTs, boolean wasSynced) Convert rollback action metadata to metadata table records.
static Map<MetadataPartitionType,HoodieData<HoodieRecord>>	convertMetadataToRecords(HoodieEngineContext engineContext, HoodieCleanMetadata cleanMetadata, MetadataRecordsGenerationParams recordsGenerationParams, String instantTime) Convert the clean action to metadata records.
static Map<MetadataPartitionType,HoodieData<HoodieRecord>>	convertMetadataToRecords(HoodieEngineContext context, HoodieCommitMetadata commitMetadata, String instantTime, MetadataRecordsGenerationParams recordsGenerationParams) Convert commit action to metadata records for the enabled partition types.
static void	deleteMetadataPartition(String basePath, HoodieEngineContext context, MetadataPartitionType partitionType) Deletes the metadata partition from the file system.
static void	deleteMetadataTable(String basePath, HoodieEngineContext context) Delete the metadata table for the dataset.
static HoodieTableFileSystemView	getFileSystemView(HoodieTableMetaClient metaClient) Get metadata table file system view.
static Set<String>	getInflightAndCompletedMetadataPartitions(HoodieTableConfig tableConfig)
static Set<String>	getInflightMetadataPartitions(HoodieTableConfig tableConfig)
static int	getPartitionFileGroupCount(MetadataPartitionType partitionType, Option<HoodieTableMetaClient> metaClient, Option<HoodieTableFileSystemView> fsView, HoodieMetadataConfig metadataConfig, boolean isBootstrapCompleted) Get file group count for a metadata table partition.
static String	getPartitionIdentifier(String relativePartitionPath) Returns partition name for the given path.
static List<FileSlice>	getPartitionLatestFileSlices(HoodieTableMetaClient metaClient, Option<HoodieTableFileSystemView> fsView, String partition) Get the latest file slices for a Metadata Table partition.
static List<FileSlice>	getPartitionLatestFileSlicesIncludingInflight(HoodieTableMetaClient metaClient, Option<HoodieTableFileSystemView> fileSystemView, String partition) Get the latest file slices for a given partition including the inflight ones.
static List<FileSlice>	getPartitionLatestMergedFileSlices(HoodieTableMetaClient metaClient, HoodieTableFileSystemView fsView, String partition) Get the latest file slices for a Metadata Table partition.
static boolean	isFilesPartitionAvailable(HoodieTableMetaClient metaClient) Returns whether the files partition of metadata table is ready for read.
static int	mapRecordKeyToFileGroupIndex(String recordKey, int numFileGroups) Map a record key to a file group in partition of interest.
static boolean	metadataPartitionExists(String basePath, HoodieEngineContext context, MetadataPartitionType partitionType) Check if the given metadata partition exists.
static BigDecimal	tryUpcastDecimal(BigDecimal value, org.apache.avro.LogicalTypes.Decimal decimal) Does an upcast for BigDecimal instance to align it with scale/precision expected by the LogicalTypes.Decimal Avro logical type

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PARTITION_NAME_FILES

public static final String PARTITION_NAME_FILES

See Also:

Constant Field Values

PARTITION_NAME_COLUMN_STATS

public static final String PARTITION_NAME_COLUMN_STATS

See Also:

Constant Field Values

PARTITION_NAME_BLOOM_FILTERS

public static final String PARTITION_NAME_BLOOM_FILTERS

See Also:

Constant Field Values

Constructor Detail

HoodieTableMetadataUtil

public HoodieTableMetadataUtil()

Method Detail

isFilesPartitionAvailable

public static boolean isFilesPartitionAvailable(HoodieTableMetaClient metaClient)

Returns whether the files partition of metadata table is ready for read.

Parameters:

metaClient - HoodieTableMetaClient instance.

Returns:

true if the files partition of metadata table is ready for read, based on the table config; false otherwise.

collectColumnRangeMetadata

public static Map<String,HoodieColumnRangeMetadata<Comparable>> collectColumnRangeMetadata(List<org.apache.avro.generic.IndexedRecord> records,
                                                                                           List<org.apache.avro.Schema.Field> targetFields,
                                                                                           String filePath)

Collects HoodieColumnRangeMetadata for the provided collection of records, pretending as if provided records have been persisted w/in given filePath

Parameters:

records - target records to compute column range metadata for

targetFields - columns (fields) to be collected

filePath - file path value required for HoodieColumnRangeMetadata

Returns:

map of HoodieColumnRangeMetadata for each of the provided target fields for the collection of provided records

convertColumnStatsRecordToColumnRangeMetadata

public static HoodieColumnRangeMetadata<Comparable> convertColumnStatsRecordToColumnRangeMetadata(HoodieMetadataColumnStats columnStats)

Converts instance of HoodieMetadataColumnStats to HoodieColumnRangeMetadata

deleteMetadataTable

public static void deleteMetadataTable(String basePath,
                                       HoodieEngineContext context)

Delete the metadata table for the dataset. This will be invoked during upgrade/downgrade operation during which no other process should be running.

Parameters:

basePath - base path of the dataset

context - instance of HoodieEngineContext.

deleteMetadataPartition

public static void deleteMetadataPartition(String basePath,
                                           HoodieEngineContext context,
                                           MetadataPartitionType partitionType)

Deletes the metadata partition from the file system.

Parameters:

basePath - - base path of the dataset

context - - instance of HoodieEngineContext

partitionType - - MetadataPartitionType of the partition to delete

metadataPartitionExists

public static boolean metadataPartitionExists(String basePath,
                                              HoodieEngineContext context,
                                              MetadataPartitionType partitionType)

Check if the given metadata partition exists.

Parameters:

basePath - base path of the dataset

context - instance of HoodieEngineContext.

convertMetadataToRecords

public static Map<MetadataPartitionType,HoodieData<HoodieRecord>> convertMetadataToRecords(HoodieEngineContext context,
                                                                                           HoodieCommitMetadata commitMetadata,
                                                                                           String instantTime,
                                                                                           MetadataRecordsGenerationParams recordsGenerationParams)

Convert commit action to metadata records for the enabled partition types.

Parameters:

commitMetadata - - Commit action metadata

instantTime - - Action instant time

recordsGenerationParams - - Parameters for the record generation

Returns:

Map of partition to metadata records for the commit action

convertMetadataToFilesPartitionRecords

public static List<HoodieRecord> convertMetadataToFilesPartitionRecords(HoodieCommitMetadata commitMetadata,
                                                                        String instantTime)

Finds all new files/partitions created as part of commit and creates metadata table records for them.

Parameters:

commitMetadata - - Commit action metadata

instantTime - - Commit action instant time

Returns:

List of metadata table records

convertMetadataToBloomFilterRecords

public static HoodieData<HoodieRecord> convertMetadataToBloomFilterRecords(HoodieEngineContext context,
                                                                           HoodieCommitMetadata commitMetadata,
                                                                           String instantTime,
                                                                           MetadataRecordsGenerationParams recordsGenerationParams)

Convert commit action metadata to bloom filter records.

Parameters:

context - - Engine context to use

commitMetadata - - Commit action metadata

instantTime - - Action instant time

recordsGenerationParams - - Parameters for bloom filter record generation

Returns:

HoodieData of metadata table records

convertMetadataToRecords

public static Map<MetadataPartitionType,HoodieData<HoodieRecord>> convertMetadataToRecords(HoodieEngineContext engineContext,
                                                                                           HoodieCleanMetadata cleanMetadata,
                                                                                           MetadataRecordsGenerationParams recordsGenerationParams,
                                                                                           String instantTime)

Convert the clean action to metadata records.

convertMetadataToFilesPartitionRecords

public static List<HoodieRecord> convertMetadataToFilesPartitionRecords(HoodieCleanMetadata cleanMetadata,
                                                                        String instantTime)

Finds all files that were deleted as part of a clean and creates metadata table records for them.

Parameters:

cleanMetadata -

instantTime -

Returns:

a list of metadata table records

convertMetadataToBloomFilterRecords

public static HoodieData<HoodieRecord> convertMetadataToBloomFilterRecords(HoodieCleanMetadata cleanMetadata,
                                                                           HoodieEngineContext engineContext,
                                                                           String instantTime,
                                                                           MetadataRecordsGenerationParams recordsGenerationParams)

Convert clean metadata to bloom filter index records.

Parameters:

cleanMetadata - - Clean action metadata

engineContext - - Engine context

instantTime - - Clean action instant time

recordsGenerationParams - - Parameters for bloom filter record generation

Returns:

List of bloom filter index records for the clean metadata

convertMetadataToColumnStatsRecords

public static HoodieData<HoodieRecord> convertMetadataToColumnStatsRecords(HoodieCleanMetadata cleanMetadata,
                                                                           HoodieEngineContext engineContext,
                                                                           MetadataRecordsGenerationParams recordsGenerationParams)

Convert clean metadata to column stats index records.

Parameters:

cleanMetadata - - Clean action metadata

engineContext - - Engine context

recordsGenerationParams - - Parameters for bloom filter record generation

Returns:

List of column stats index records for the clean metadata

convertMetadataToRecords

public static Map<MetadataPartitionType,HoodieData<HoodieRecord>> convertMetadataToRecords(HoodieEngineContext engineContext,
                                                                                           HoodieActiveTimeline metadataTableTimeline,
                                                                                           HoodieRestoreMetadata restoreMetadata,
                                                                                           MetadataRecordsGenerationParams recordsGenerationParams,
                                                                                           String instantTime,
                                                                                           Option<String> lastSyncTs)

Convert restore action metadata to metadata table records.

convertMetadataToRecords

public static Map<MetadataPartitionType,HoodieData<HoodieRecord>> convertMetadataToRecords(HoodieEngineContext engineContext,
                                                                                           HoodieActiveTimeline metadataTableTimeline,
                                                                                           HoodieRollbackMetadata rollbackMetadata,
                                                                                           MetadataRecordsGenerationParams recordsGenerationParams,
                                                                                           String instantTime,
                                                                                           Option<String> lastSyncTs,
                                                                                           boolean wasSynced)

Convert rollback action metadata to metadata table records.

getPartitionIdentifier

public static String getPartitionIdentifier(@Nonnull
                                            String relativePartitionPath)

Returns partition name for the given path.

convertFilesToBloomFilterRecords

public static HoodieData<HoodieRecord> convertFilesToBloomFilterRecords(HoodieEngineContext engineContext,
                                                                        Map<String,List<String>> partitionToDeletedFiles,
                                                                        Map<String,Map<String,Long>> partitionToAppendedFiles,
                                                                        MetadataRecordsGenerationParams recordsGenerationParams,
                                                                        String instantTime)

Convert added and deleted files metadata to bloom filter index records.

convertFilesToColumnStatsRecords

public static HoodieData<HoodieRecord> convertFilesToColumnStatsRecords(HoodieEngineContext engineContext,
                                                                        Map<String,List<String>> partitionToDeletedFiles,
                                                                        Map<String,Map<String,Long>> partitionToAppendedFiles,
                                                                        MetadataRecordsGenerationParams recordsGenerationParams)

Convert added and deleted action metadata to column stats index records.

mapRecordKeyToFileGroupIndex

public static int mapRecordKeyToFileGroupIndex(String recordKey,
                                               int numFileGroups)

Map a record key to a file group in partition of interest.

Note: For hashing, the algorithm is same as String.hashCode() but is being defined here as hashCode() implementation is not guaranteed by the JVM to be consistent across JVM versions and implementations.

Parameters:

recordKey - record key for which the file group index is looked up for.

Returns:

An integer hash of the given string

getPartitionLatestMergedFileSlices

public static List<FileSlice> getPartitionLatestMergedFileSlices(HoodieTableMetaClient metaClient,
                                                                 HoodieTableFileSystemView fsView,
                                                                 String partition)

Get the latest file slices for a Metadata Table partition. If the file slice is because of pending compaction instant, then merge the file slice with the one just before the compaction instant time. The list of file slices returned is sorted in the correct order of file group name.

Parameters:

metaClient - Instance of HoodieTableMetaClient.

fsView - Metadata table filesystem view.

partition - The name of the partition whose file groups are to be loaded.

Returns:

List of latest file slices for all file groups in a given partition.

getPartitionLatestFileSlices

public static List<FileSlice> getPartitionLatestFileSlices(HoodieTableMetaClient metaClient,
                                                           Option<HoodieTableFileSystemView> fsView,
                                                           String partition)

Get the latest file slices for a Metadata Table partition. The list of file slices returned is sorted in the correct order of file group name.

Parameters:

metaClient - - Instance of HoodieTableMetaClient.

fsView - - Metadata table filesystem view

partition - - The name of the partition whose file groups are to be loaded.

Returns:

List of latest file slices for all file groups in a given partition.

getFileSystemView

public static HoodieTableFileSystemView getFileSystemView(HoodieTableMetaClient metaClient)

Get metadata table file system view.

Parameters:

metaClient - - Metadata table meta client

Returns:

Filesystem view for the metadata table

getPartitionLatestFileSlicesIncludingInflight

public static List<FileSlice> getPartitionLatestFileSlicesIncludingInflight(HoodieTableMetaClient metaClient,
                                                                            Option<HoodieTableFileSystemView> fileSystemView,
                                                                            String partition)

Get the latest file slices for a given partition including the inflight ones.

Parameters:

metaClient - - instance of HoodieTableMetaClient

fileSystemView - - hoodie table file system view, which will be fetched from meta client if not already present

partition - - name of the partition whose file groups are to be loaded

Returns:

convertMetadataToColumnStatsRecords

public static HoodieData<HoodieRecord> convertMetadataToColumnStatsRecords(HoodieCommitMetadata commitMetadata,
                                                                           HoodieEngineContext engineContext,
                                                                           MetadataRecordsGenerationParams recordsGenerationParams)

getPartitionFileGroupCount

public static int getPartitionFileGroupCount(MetadataPartitionType partitionType,
                                             Option<HoodieTableMetaClient> metaClient,
                                             Option<HoodieTableFileSystemView> fsView,
                                             HoodieMetadataConfig metadataConfig,
                                             boolean isBootstrapCompleted)

Get file group count for a metadata table partition.

Parameters:

partitionType - - Metadata table partition type

metaClient - - Metadata table meta client

fsView - - Filesystem view

metadataConfig - - Metadata config

isBootstrapCompleted - - Is bootstrap completed for the metadata table

Returns:

File group count for the requested metadata partition type

tryUpcastDecimal

public static BigDecimal tryUpcastDecimal(BigDecimal value,
                                          org.apache.avro.LogicalTypes.Decimal decimal)

Does an upcast for BigDecimal instance to align it with scale/precision expected by the LogicalTypes.Decimal Avro logical type

getInflightMetadataPartitions

public static Set<String> getInflightMetadataPartitions(HoodieTableConfig tableConfig)

getInflightAndCompletedMetadataPartitions

public static Set<String> getInflightAndCompletedMetadataPartitions(HoodieTableConfig tableConfig)