org.apache.hudi.metadata

Interface HoodieTableMetadata

All Superinterfaces:

AutoCloseable, Serializable

All Known Implementing Classes:

AbstractHoodieTableMetadata, BaseTableMetadata, FileSystemBackedTableMetadata, HoodieBackedTableMetadata

public interface HoodieTableMetadata
extends Serializable, AutoCloseable

Interface that supports querying various pieces of metadata about a hudi table.

Field Summary

Fields

Modifier and Type	Field and Description
static String	EMPTY_PARTITION_NAME
static String	METADATA_TABLE_NAME_SUFFIX
static String	NON_PARTITIONED_NAME
static String	RECORDKEY_PARTITION_LIST
static String	SOLO_COMMIT_TIMESTAMP Timestamp for a commit when the base dataset had not had any commits yet.

Method Summary

Modifier and Type	Method and Description
static HoodieTableMetadata	create(HoodieEngineContext engineContext, HoodieStorage storage, HoodieMetadataConfig metadataConfig, String datasetBasePath)
static HoodieTableMetadata	create(HoodieEngineContext engineContext, HoodieStorage storage, HoodieMetadataConfig metadataConfig, String datasetBasePath, boolean reuse)
static FileSystemBackedTableMetadata	createFSBackedTableMetadata(HoodieEngineContext engineContext, HoodieStorage storage, HoodieMetadataConfig metadataConfig, String datasetBasePath)
static HoodieBackedTableMetadata	createHoodieBackedTableMetadata(HoodieEngineContext engineContext, HoodieStorage storage, HoodieMetadataConfig metadataConfig, String datasetBasePath, boolean reuse)
List<StoragePathInfo>	getAllFilesInPartition(StoragePath partitionPath) Fetch all the files at the given partition path, per the latest snapshot of the metadata.
Map<String,List<StoragePathInfo>>	getAllFilesInPartitions(Collection<String> partitionPaths) Fetch all files for given partition paths.
List<String>	getAllPartitionPaths() Fetch list of all partition paths, per the latest snapshot of the metadata.
Option<BloomFilter>	getBloomFilter(String partitionName, String fileName) Get the bloom filter for the FileID from the metadata table.
Map<Pair<String,String>,BloomFilter>	getBloomFilters(List<Pair<String,String>> partitionNameFileNameList) Get bloom filters for files from the metadata table index.
Map<Pair<String,String>,HoodieMetadataColumnStats>	getColumnStats(List<Pair<String,String>> partitionNameFileNameList, String columnName) Get column stats for files from the metadata table index.
static String	getDatasetBasePath(String metadataTableBasePath) Return the base path of the dataset.
static String	getDataTableBasePathFromMetadataTable(String metadataTableBasePath) Returns the base path of the Dataset provided the base-path of the Metadata Table of this Dataset
Option<String>	getLatestCompactionTime() Returns the timestamp of the latest compaction.
static StoragePath	getMetadataTableBasePath(StoragePath dataTableBasePath) Return the base-path of the Metadata Table for the given Dataset identified by base-path
static String	getMetadataTableBasePath(String dataTableBasePath) Return the base-path of the Metadata Table for the given Dataset identified by base-path
int	getNumFileGroupsForPartition(MetadataPartitionType partition) Returns the number of shards in a metadata table partition.
List<String>	getPartitionPathWithPathPrefixes(List<String> relativePathPrefixes) Fetches all partition paths that are the sub-directories of the list of provided (relative) paths.
List<String>	getPartitionPathWithPathPrefixUsingFilterExpression(List<String> relativePathPrefixes, Types.RecordType partitionFields, Expression expression) Retrieve the paths of partitions under the provided sub-directories, and try to filter these partitions using the provided Expression.
HoodieData<HoodieRecord<HoodieMetadataPayload>>	getRecordsByKeyPrefixes(List<String> keyPrefixes, String partitionName, boolean shouldLoadInMemory) Fetch records by key prefixes.
Option<String>	getSyncedInstantTime() Get the instant time to which the metadata is synced w.r.t data timeline.
static boolean	isMetadataTable(String basePath) Returns True if the given path contains a metadata table.
Map<String,HoodieRecordGlobalLocation>	readRecordIndex(List<String> recordKeys) Returns the location of record keys which are found in the record index.
void	reset() Clear the states of the table metadata.

Methods inherited from interface java.lang.AutoCloseable

close

Field Detail

METADATA_TABLE_NAME_SUFFIX

static final String METADATA_TABLE_NAME_SUFFIX

See Also:

Constant Field Values

SOLO_COMMIT_TIMESTAMP

static final String SOLO_COMMIT_TIMESTAMP

Timestamp for a commit when the base dataset had not had any commits yet. this is < than even HoodieTimeline.INIT_INSTANT_TS, such that the metadata table can be prepped even before bootstrap is done.

See Also:

Constant Field Values

RECORDKEY_PARTITION_LIST

static final String RECORDKEY_PARTITION_LIST

See Also:

Constant Field Values

NON_PARTITIONED_NAME

static final String NON_PARTITIONED_NAME

See Also:

Constant Field Values

EMPTY_PARTITION_NAME

static final String EMPTY_PARTITION_NAME

See Also:

Constant Field Values

Method Detail

getMetadataTableBasePath

static String getMetadataTableBasePath(String dataTableBasePath)

Return the base-path of the Metadata Table for the given Dataset identified by base-path

getMetadataTableBasePath

static StoragePath getMetadataTableBasePath(StoragePath dataTableBasePath)

Return the base-path of the Metadata Table for the given Dataset identified by base-path

getDataTableBasePathFromMetadataTable

static String getDataTableBasePathFromMetadataTable(String metadataTableBasePath)

Returns the base path of the Dataset provided the base-path of the Metadata Table of this Dataset

getDatasetBasePath

static String getDatasetBasePath(String metadataTableBasePath)

Return the base path of the dataset.

Parameters:

metadataTableBasePath - The base path of the metadata table

isMetadataTable

static boolean isMetadataTable(String basePath)

Returns True if the given path contains a metadata table.

Parameters:

basePath - The base path to check

create

static HoodieTableMetadata create(HoodieEngineContext engineContext,
                                  HoodieStorage storage,
                                  HoodieMetadataConfig metadataConfig,
                                  String datasetBasePath)

create

static HoodieTableMetadata create(HoodieEngineContext engineContext,
                                  HoodieStorage storage,
                                  HoodieMetadataConfig metadataConfig,
                                  String datasetBasePath,
                                  boolean reuse)

createFSBackedTableMetadata

static FileSystemBackedTableMetadata createFSBackedTableMetadata(HoodieEngineContext engineContext,
                                                                 HoodieStorage storage,
                                                                 HoodieMetadataConfig metadataConfig,
                                                                 String datasetBasePath)

createHoodieBackedTableMetadata

static HoodieBackedTableMetadata createHoodieBackedTableMetadata(HoodieEngineContext engineContext,
                                                                 HoodieStorage storage,
                                                                 HoodieMetadataConfig metadataConfig,
                                                                 String datasetBasePath,
                                                                 boolean reuse)

getAllFilesInPartition

List<StoragePathInfo> getAllFilesInPartition(StoragePath partitionPath)
                                      throws IOException

Fetch all the files at the given partition path, per the latest snapshot of the metadata.

Throws:

IOException

getPartitionPathWithPathPrefixUsingFilterExpression

List<String> getPartitionPathWithPathPrefixUsingFilterExpression(List<String> relativePathPrefixes,
                                                                 Types.RecordType partitionFields,
                                                                 Expression expression)
                                                          throws IOException

Retrieve the paths of partitions under the provided sub-directories, and try to filter these partitions using the provided Expression.

Throws:

IOException

getPartitionPathWithPathPrefixes

List<String> getPartitionPathWithPathPrefixes(List<String> relativePathPrefixes)
                                       throws IOException

Fetches all partition paths that are the sub-directories of the list of provided (relative) paths.

E.g., Table has partition 4 partitions: year=2022/month=08/day=30, year=2022/month=08/day=31, year=2022/month=07/day=03, year=2022/month=07/day=04 The relative path "year=2022" returns all partitions, while the relative path "year=2022/month=07" returns only two partitions.

Throws:

IOException

getAllPartitionPaths

List<String> getAllPartitionPaths()
                           throws IOException

Fetch list of all partition paths, per the latest snapshot of the metadata.

Throws:

IOException

getAllFilesInPartitions

Map<String,List<StoragePathInfo>> getAllFilesInPartitions(Collection<String> partitionPaths)
                                                   throws IOException

Fetch all files for given partition paths. NOTE: Absolute partition paths are expected here

Throws:

IOException

getBloomFilter

Option<BloomFilter> getBloomFilter(String partitionName,
                                   String fileName)
                            throws HoodieMetadataException

Get the bloom filter for the FileID from the metadata table.

Parameters:

partitionName - - Partition name

fileName - - File name for which bloom filter needs to be retrieved

Returns:

BloomFilter if available, otherwise empty

Throws:

HoodieMetadataException

getBloomFilters

Map<Pair<String,String>,BloomFilter> getBloomFilters(List<Pair<String,String>> partitionNameFileNameList)
                                              throws HoodieMetadataException

Get bloom filters for files from the metadata table index.

Parameters:

partitionNameFileNameList - - List of partition and file name pair for which bloom filters need to be retrieved

Returns:

Map of partition file name pair to its bloom filter

Throws:

HoodieMetadataException

getColumnStats

Map<Pair<String,String>,HoodieMetadataColumnStats> getColumnStats(List<Pair<String,String>> partitionNameFileNameList,
                                                                  String columnName)
                                                           throws HoodieMetadataException

Get column stats for files from the metadata table index.

Parameters:

partitionNameFileNameList - - List of partition and file name pair for which bloom filters need to be retrieved

columnName - - Column name for which stats are needed

Returns:

Map of partition and file name pair to its column stats

Throws:

HoodieMetadataException

readRecordIndex

Map<String,HoodieRecordGlobalLocation> readRecordIndex(List<String> recordKeys)

Returns the location of record keys which are found in the record index. Records that are not found are ignored and wont be part of map object that is returned.

getRecordsByKeyPrefixes

HoodieData<HoodieRecord<HoodieMetadataPayload>> getRecordsByKeyPrefixes(List<String> keyPrefixes,
                                                                        String partitionName,
                                                                        boolean shouldLoadInMemory)

Fetch records by key prefixes. Key prefix passed is expected to match the same prefix as stored in Metadata table partitions. For eg, in case of col stats partition, actual keys in metadata partition is encoded values of column name, partition name and file name. So, key prefixes passed to this method is expected to be encoded already.

Parameters:

keyPrefixes - list of key prefixes for which interested records are looked up for.

partitionName - partition name in metadata table where the records are looked up for.

Returns:

HoodieData of HoodieRecords with records matching the passed in key prefixes.

getSyncedInstantTime

Option<String> getSyncedInstantTime()

Get the instant time to which the metadata is synced w.r.t data timeline.

getLatestCompactionTime

Option<String> getLatestCompactionTime()

Returns the timestamp of the latest compaction.

reset

void reset()

Clear the states of the table metadata.

getNumFileGroupsForPartition

int getNumFileGroupsForPartition(MetadataPartitionType partition)

Returns the number of shards in a metadata table partition.