org.apache.hudi.metadata

Class HoodieBackedTableMetadata

java.lang.Object

org.apache.hudi.metadata.BaseTableMetadata

org.apache.hudi.metadata.HoodieBackedTableMetadata

All Implemented Interfaces:

Serializable, AutoCloseable, HoodieTableMetadata

public class HoodieBackedTableMetadata
extends BaseTableMetadata

Table metadata provided by an internal DFS backed Hudi metadata table.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.hudi.metadata.BaseTableMetadata

BUFFER_SIZE, dataBasePath, dataMetaClient, engineContext, hadoopConf, isBloomFilterIndexEnabled, isColumnStatsIndexEnabled, isMetadataTableEnabled, MAX_MEMORY_SIZE_IN_BYTES, metadataConfig, metrics, spillableMapDirectory

Fields inherited from interface org.apache.hudi.metadata.HoodieTableMetadata

EMPTY_PARTITION_NAME, METADATA_TABLE_NAME_SUFFIX, NON_PARTITIONED_NAME, RECORDKEY_PARTITION_LIST, SOLO_COMMIT_TIMESTAMP

Constructor Summary

Constructors

Constructor and Description
HoodieBackedTableMetadata(HoodieEngineContext engineContext, HoodieMetadataConfig metadataConfig, String datasetBasePath, String spillableMapDirectory)
HoodieBackedTableMetadata(HoodieEngineContext engineContext, HoodieMetadataConfig metadataConfig, String datasetBasePath, String spillableMapDirectory, boolean reuse)

Method Summary

Modifier and Type	Method and Description
void	close()
boolean	enabled()
SerializableConfiguration	getHadoopConf()
Option<String>	getLatestCompactionTime() Returns the timestamp of the latest compaction.
Pair<HoodieMetadataMergedLogRecordReader,Long>	getLogRecordScanner(List<HoodieLogFile> logFiles, String partitionName, Option<Boolean> allowFullScanOverride)
HoodieTableMetaClient	getMetadataMetaClient()
protected Option<HoodieRecord<HoodieMetadataPayload>>	getRecordByKey(String key, String partitionName)
HoodieData<HoodieRecord<HoodieMetadataPayload>>	getRecordsByKeyPrefixes(List<String> keyPrefixes, String partitionName, boolean shouldLoadInMemory) Fetch records by key prefixes.
List<Pair<String,Option<HoodieRecord<HoodieMetadataPayload>>>>	getRecordsByKeys(List<String> keys, String partitionName)
Option<String>	getSyncedInstantTime() Get the instant time to which the metadata is synced w.r.t data timeline.
void	reset() Clear the states of the table metadata.
Map<String,String>	stats()

Methods inherited from class org.apache.hudi.metadata.BaseTableMetadata

fetchAllPartitionPaths, getAllFilesInPartition, getAllFilesInPartitions, getAllPartitionPaths, getBloomFilter, getBloomFilters, getColumnStats, getEngineContext, getLatestDataInstantTime, getMetadataConfig

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hudi.metadata.HoodieTableMetadata

create, create, createFSBackedTableMetadata, createHoodieBackedTableMetadata, getDatasetBasePath, getDataTableBasePathFromMetadataTable, getMetadataTableBasePath, isMetadataTable

Constructor Detail

HoodieBackedTableMetadata

public HoodieBackedTableMetadata(HoodieEngineContext engineContext,
                                 HoodieMetadataConfig metadataConfig,
                                 String datasetBasePath,
                                 String spillableMapDirectory)

HoodieBackedTableMetadata

public HoodieBackedTableMetadata(HoodieEngineContext engineContext,
                                 HoodieMetadataConfig metadataConfig,
                                 String datasetBasePath,
                                 String spillableMapDirectory,
                                 boolean reuse)

Method Detail

getRecordByKey

protected Option<HoodieRecord<HoodieMetadataPayload>> getRecordByKey(String key,
                                                                     String partitionName)

Specified by:

getRecordByKey in class BaseTableMetadata

getRecordsByKeyPrefixes

public HoodieData<HoodieRecord<HoodieMetadataPayload>> getRecordsByKeyPrefixes(List<String> keyPrefixes,
                                                                               String partitionName,
                                                                               boolean shouldLoadInMemory)

Description copied from interface: HoodieTableMetadata

Fetch records by key prefixes. Key prefix passed is expected to match the same prefix as stored in Metadata table partitions. For eg, in case of col stats partition, actual keys in metadata partition is encoded values of column name, partition name and file name. So, key prefixes passed to this method is expected to be encoded already.

Parameters:

keyPrefixes - list of key prefixes for which interested records are looked up for.

partitionName - partition name in metadata table where the records are looked up for.

Returns:

HoodieData of HoodieRecords with records matching the passed in key prefixes.

getRecordsByKeys

public List<Pair<String,Option<HoodieRecord<HoodieMetadataPayload>>>> getRecordsByKeys(List<String> keys,
                                                                                       String partitionName)

Specified by:

getRecordsByKeys in class BaseTableMetadata

getLogRecordScanner

public Pair<HoodieMetadataMergedLogRecordReader,Long> getLogRecordScanner(List<HoodieLogFile> logFiles,
                                                                          String partitionName,
                                                                          Option<Boolean> allowFullScanOverride)

close

public void close()

enabled

public boolean enabled()

getHadoopConf

public SerializableConfiguration getHadoopConf()

getMetadataMetaClient

public HoodieTableMetaClient getMetadataMetaClient()

stats

public Map<String,String> stats()

getSyncedInstantTime

public Option<String> getSyncedInstantTime()

Description copied from interface: HoodieTableMetadata

Get the instant time to which the metadata is synced w.r.t data timeline.

getLatestCompactionTime

public Option<String> getLatestCompactionTime()

Description copied from interface: HoodieTableMetadata

Returns the timestamp of the latest compaction.

reset

public void reset()

Description copied from interface: HoodieTableMetadata

Clear the states of the table metadata.