org.apache.hudi.index.bucket

Class ConsistentBucketIndexUtils

java.lang.Object

org.apache.hudi.index.bucket.ConsistentBucketIndexUtils

public class ConsistentBucketIndexUtils
extends Object

Utilities class for consistent bucket index metadata management.

Constructor Summary

Constructors

Constructor and Description
ConsistentBucketIndexUtils()

Method Summary

Modifier and Type	Method and Description
static Map<String,Map<String,Integer>>	generatePartitionToFileIdPfxIdxMap(Map<String,ConsistentBucketIdentifier> partitionToIdentifier) Initialize fileIdPfx for each data partition.
static Option<HoodieConsistentHashingMetadata>	loadMetadata(HoodieTable table, String partition) Loads hashing metadata of the given partition, if it does not exist, returns empty.
static HoodieConsistentHashingMetadata	loadOrCreateMetadata(HoodieTable table, String partition, int numBuckets) Loads hashing metadata of the given partition, if it does not exist, creates a new one (also persist it into storage).
static boolean	saveMetadata(HoodieTable table, HoodieConsistentHashingMetadata metadata, boolean overwrite) Saves the metadata into storage

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ConsistentBucketIndexUtils

public ConsistentBucketIndexUtils()

Method Detail

loadOrCreateMetadata

public static HoodieConsistentHashingMetadata loadOrCreateMetadata(HoodieTable table,
                                                                   String partition,
                                                                   int numBuckets)

Loads hashing metadata of the given partition, if it does not exist, creates a new one (also persist it into storage).

NOTE: When creating a new hashing metadata, the content will always be the same for the same partition. It means when multiple writer are trying to initialize metadata for the same partition, no lock or synchronization is necessary as they are creating the file with the same content.

Parameters:

table - Hoodie table

partition - Table partition

numBuckets - Default bucket number

Returns:

Consistent hashing metadata

loadMetadata

public static Option<HoodieConsistentHashingMetadata> loadMetadata(HoodieTable table,
                                                                   String partition)

Loads hashing metadata of the given partition, if it does not exist, returns empty.

Parameters:

table - Hoodie table

partition - Table partition

Returns:

Consistent hashing metadata or empty if it does not exist

saveMetadata

public static boolean saveMetadata(HoodieTable table,
                                   HoodieConsistentHashingMetadata metadata,
                                   boolean overwrite)

Saves the metadata into storage

Parameters:

table - Hoodie table

metadata - Hashing metadata to be saved

overwrite - Whether to overwrite existing metadata

Returns:

true if the metadata is saved successfully

generatePartitionToFileIdPfxIdxMap

public static Map<String,Map<String,Integer>> generatePartitionToFileIdPfxIdxMap(Map<String,ConsistentBucketIdentifier> partitionToIdentifier)

Initialize fileIdPfx for each data partition. Specifically, the following fields is constructed: - fileIdPfxList: the Nth element corresponds to the Nth data partition, indicating its fileIdPfx - partitionToFileIdPfxIdxMap (return value): (table partition) -> (fileIdPfx -> idx) mapping

Parameters:

partitionToIdentifier - Mapping from table partition to bucket identifier