org.apache.hudi.metadata

Class BaseTableMetadata

java.lang.Object

org.apache.hudi.metadata.BaseTableMetadata

All Implemented Interfaces:

Serializable, AutoCloseable, HoodieTableMetadata

Direct Known Subclasses:

HoodieBackedTableMetadata

public abstract class BaseTableMetadata
extends Object
implements HoodieTableMetadata

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static int	BUFFER_SIZE
protected SerializablePath	dataBasePath
protected HoodieTableMetaClient	dataMetaClient
protected HoodieEngineContext	engineContext
protected SerializableConfiguration	hadoopConf
protected boolean	isBloomFilterIndexEnabled
protected boolean	isColumnStatsIndexEnabled
protected boolean	isMetadataTableEnabled
static long	MAX_MEMORY_SIZE_IN_BYTES
protected HoodieMetadataConfig	metadataConfig
protected Option<HoodieMetadataMetrics>	metrics
protected String	spillableMapDirectory

Fields inherited from interface org.apache.hudi.metadata.HoodieTableMetadata

EMPTY_PARTITION_NAME, METADATA_TABLE_NAME_SUFFIX, NON_PARTITIONED_NAME, RECORDKEY_PARTITION_LIST, SOLO_COMMIT_TIMESTAMP

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	BaseTableMetadata(HoodieEngineContext engineContext, HoodieMetadataConfig metadataConfig, String dataBasePath, String spillableMapDirectory)

Method Summary

Modifier and Type	Method and Description
protected List<String>	fetchAllPartitionPaths() Returns a list of all partitions.
org.apache.hadoop.fs.FileStatus[]	getAllFilesInPartition(org.apache.hadoop.fs.Path partitionPath) Return the list of files in a partition.
Map<String,org.apache.hadoop.fs.FileStatus[]>	getAllFilesInPartitions(List<String> partitions) Fetch all files for given partition paths.
List<String>	getAllPartitionPaths() Return the list of partitions in the dataset.
Option<BloomFilter>	getBloomFilter(String partitionName, String fileName) Get the bloom filter for the FileID from the metadata table.
Map<Pair<String,String>,BloomFilter>	getBloomFilters(List<Pair<String,String>> partitionNameFileNameList) Get bloom filters for files from the metadata table index.
Map<Pair<String,String>,HoodieMetadataColumnStats>	getColumnStats(List<Pair<String,String>> partitionNameFileNameList, String columnName) Get column stats for files from the metadata table index.
protected HoodieEngineContext	getEngineContext()
protected String	getLatestDataInstantTime()
HoodieMetadataConfig	getMetadataConfig()
protected abstract Option<HoodieRecord<HoodieMetadataPayload>>	getRecordByKey(String key, String partitionName)
abstract List<Pair<String,Option<HoodieRecord<HoodieMetadataPayload>>>>	getRecordsByKeys(List<String> key, String partitionName)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hudi.metadata.HoodieTableMetadata

create, create, createFSBackedTableMetadata, createHoodieBackedTableMetadata, getDatasetBasePath, getDataTableBasePathFromMetadataTable, getLatestCompactionTime, getMetadataTableBasePath, getRecordsByKeyPrefixes, getSyncedInstantTime, isMetadataTable, reset

Methods inherited from interface java.lang.AutoCloseable

close

Field Detail

MAX_MEMORY_SIZE_IN_BYTES

public static final long MAX_MEMORY_SIZE_IN_BYTES

See Also:

Constant Field Values

BUFFER_SIZE

public static final int BUFFER_SIZE

See Also:

Constant Field Values

engineContext

protected final transient HoodieEngineContext engineContext

hadoopConf

protected final SerializableConfiguration hadoopConf

dataBasePath

protected final SerializablePath dataBasePath

dataMetaClient

protected final HoodieTableMetaClient dataMetaClient

metrics

protected final Option<HoodieMetadataMetrics> metrics

metadataConfig

protected final HoodieMetadataConfig metadataConfig

spillableMapDirectory

protected final String spillableMapDirectory

isMetadataTableEnabled

protected boolean isMetadataTableEnabled

isBloomFilterIndexEnabled

protected boolean isBloomFilterIndexEnabled

isColumnStatsIndexEnabled

protected boolean isColumnStatsIndexEnabled

Constructor Detail

BaseTableMetadata

protected BaseTableMetadata(HoodieEngineContext engineContext,
                            HoodieMetadataConfig metadataConfig,
                            String dataBasePath,
                            String spillableMapDirectory)

Method Detail

getAllPartitionPaths

public List<String> getAllPartitionPaths()
                                  throws IOException

Return the list of partitions in the dataset.

If the Metadata Table is enabled, the listing is retrieved from the stored metadata. Otherwise, the list of partitions is retrieved directly from the underlying FileSystem.

On any errors retrieving the listing from the metadata, defaults to using the file system listings.

Specified by:

getAllPartitionPaths in interface HoodieTableMetadata

Throws:

IOException

getAllFilesInPartition

public org.apache.hadoop.fs.FileStatus[] getAllFilesInPartition(org.apache.hadoop.fs.Path partitionPath)
                                                         throws IOException

Return the list of files in a partition.

If the Metadata Table is enabled, the listing is retrieved from the stored metadata. Otherwise, the list of partitions is retrieved directly from the underlying FileSystem.

On any errors retrieving the listing from the metadata, defaults to using the file system listings.

Specified by:

getAllFilesInPartition in interface HoodieTableMetadata

Parameters:

partitionPath - The absolute path of the partition to list

Throws:

IOException

getAllFilesInPartitions

public Map<String,org.apache.hadoop.fs.FileStatus[]> getAllFilesInPartitions(List<String> partitions)
                                                                      throws IOException

Description copied from interface: HoodieTableMetadata

Fetch all files for given partition paths.

Specified by:

getAllFilesInPartitions in interface HoodieTableMetadata

Throws:

IOException

getBloomFilter

public Option<BloomFilter> getBloomFilter(String partitionName,
                                          String fileName)
                                   throws HoodieMetadataException

Description copied from interface: HoodieTableMetadata

Get the bloom filter for the FileID from the metadata table.

Specified by:

getBloomFilter in interface HoodieTableMetadata

Parameters:

partitionName - - Partition name

fileName - - File name for which bloom filter needs to be retrieved

Returns:

BloomFilter if available, otherwise empty

Throws:

HoodieMetadataException

getBloomFilters

public Map<Pair<String,String>,BloomFilter> getBloomFilters(List<Pair<String,String>> partitionNameFileNameList)
                                                     throws HoodieMetadataException

Description copied from interface: HoodieTableMetadata

Get bloom filters for files from the metadata table index.

Specified by:

getBloomFilters in interface HoodieTableMetadata

Parameters:

partitionNameFileNameList - - List of partition and file name pair for which bloom filters need to be retrieved

Returns:

Map of partition file name pair to its bloom filter

Throws:

HoodieMetadataException

getColumnStats

public Map<Pair<String,String>,HoodieMetadataColumnStats> getColumnStats(List<Pair<String,String>> partitionNameFileNameList,
                                                                         String columnName)
                                                                  throws HoodieMetadataException

Description copied from interface: HoodieTableMetadata

Get column stats for files from the metadata table index.

Specified by:

getColumnStats in interface HoodieTableMetadata

Parameters:

partitionNameFileNameList - - List of partition and file name pair for which bloom filters need to be retrieved

columnName - - Column name for which stats are needed

Returns:

Map of partition and file name pair to its column stats

Throws:

HoodieMetadataException

fetchAllPartitionPaths

protected List<String> fetchAllPartitionPaths()

Returns a list of all partitions.

getRecordByKey

protected abstract Option<HoodieRecord<HoodieMetadataPayload>> getRecordByKey(String key,
                                                                              String partitionName)

getRecordsByKeys

public abstract List<Pair<String,Option<HoodieRecord<HoodieMetadataPayload>>>> getRecordsByKeys(List<String> key,
                                                                                                String partitionName)

getEngineContext

protected HoodieEngineContext getEngineContext()

getMetadataConfig

public HoodieMetadataConfig getMetadataConfig()

getLatestDataInstantTime

protected String getLatestDataInstantTime()