org.apache.hudi.metadata

Class FileSystemBackedTableMetadata

java.lang.Object

org.apache.hudi.metadata.FileSystemBackedTableMetadata

All Implemented Interfaces:

Serializable, AutoCloseable, HoodieTableMetadata

public class FileSystemBackedTableMetadata
extends Object
implements HoodieTableMetadata

See Also:

Serialized Form

Field Summary

Fields inherited from interface org.apache.hudi.metadata.HoodieTableMetadata

EMPTY_PARTITION_NAME, METADATA_TABLE_NAME_SUFFIX, NON_PARTITIONED_NAME, RECORDKEY_PARTITION_LIST, SOLO_COMMIT_TIMESTAMP

Constructor Summary

Constructors

Constructor and Description
FileSystemBackedTableMetadata(HoodieEngineContext engineContext, SerializableConfiguration conf, String datasetBasePath, boolean assumeDatePartitioning)

Method Summary

Modifier and Type	Method and Description
void	close()
org.apache.hadoop.fs.FileStatus[]	getAllFilesInPartition(org.apache.hadoop.fs.Path partitionPath) Fetch all the files at the given partition path, per the latest snapshot of the metadata.
Map<String,org.apache.hadoop.fs.FileStatus[]>	getAllFilesInPartitions(List<String> partitionPaths) Fetch all files for given partition paths.
List<String>	getAllPartitionPaths() Fetch list of all partition paths, per the latest snapshot of the metadata.
Option<BloomFilter>	getBloomFilter(String partitionName, String fileName) Get the bloom filter for the FileID from the metadata table.
Map<Pair<String,String>,BloomFilter>	getBloomFilters(List<Pair<String,String>> partitionNameFileNameList) Get bloom filters for files from the metadata table index.
Map<Pair<String,String>,HoodieMetadataColumnStats>	getColumnStats(List<Pair<String,String>> partitionNameFileNameList, String columnName) Get column stats for files from the metadata table index.
Option<String>	getLatestCompactionTime() Returns the timestamp of the latest compaction.
HoodieData<HoodieRecord<HoodieMetadataPayload>>	getRecordsByKeyPrefixes(List<String> keyPrefixes, String partitionName, boolean shouldLoadInMemory) Fetch records by key prefixes.
Option<String>	getSyncedInstantTime() Get the instant time to which the metadata is synced w.r.t data timeline.
void	reset() Clear the states of the table metadata.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hudi.metadata.HoodieTableMetadata

create, create, createFSBackedTableMetadata, createHoodieBackedTableMetadata, getDatasetBasePath, getDataTableBasePathFromMetadataTable, getMetadataTableBasePath, isMetadataTable

Constructor Detail

FileSystemBackedTableMetadata

public FileSystemBackedTableMetadata(HoodieEngineContext engineContext,
                                     SerializableConfiguration conf,
                                     String datasetBasePath,
                                     boolean assumeDatePartitioning)

Method Detail

getAllFilesInPartition

public org.apache.hadoop.fs.FileStatus[] getAllFilesInPartition(org.apache.hadoop.fs.Path partitionPath)
                                                         throws IOException

Description copied from interface: HoodieTableMetadata

Fetch all the files at the given partition path, per the latest snapshot of the metadata.

Specified by:

getAllFilesInPartition in interface HoodieTableMetadata

Throws:

IOException

getAllPartitionPaths

public List<String> getAllPartitionPaths()
                                  throws IOException

Description copied from interface: HoodieTableMetadata

Fetch list of all partition paths, per the latest snapshot of the metadata.

Specified by:

getAllPartitionPaths in interface HoodieTableMetadata

Throws:

IOException

getAllFilesInPartitions

public Map<String,org.apache.hadoop.fs.FileStatus[]> getAllFilesInPartitions(List<String> partitionPaths)
                                                                      throws IOException

Description copied from interface: HoodieTableMetadata

Fetch all files for given partition paths.

Specified by:

getAllFilesInPartitions in interface HoodieTableMetadata

Throws:

IOException

getSyncedInstantTime

public Option<String> getSyncedInstantTime()

Description copied from interface: HoodieTableMetadata

Get the instant time to which the metadata is synced w.r.t data timeline.

Specified by:

getSyncedInstantTime in interface HoodieTableMetadata

getLatestCompactionTime

public Option<String> getLatestCompactionTime()

Description copied from interface: HoodieTableMetadata

Returns the timestamp of the latest compaction.

Specified by:

getLatestCompactionTime in interface HoodieTableMetadata

close

public void close()
           throws Exception

Specified by:

close in interface AutoCloseable

Throws:

Exception

reset

public void reset()

Description copied from interface: HoodieTableMetadata

Clear the states of the table metadata.

Specified by:

reset in interface HoodieTableMetadata

getBloomFilter

public Option<BloomFilter> getBloomFilter(String partitionName,
                                          String fileName)
                                   throws HoodieMetadataException

Description copied from interface: HoodieTableMetadata

Get the bloom filter for the FileID from the metadata table.

Specified by:

getBloomFilter in interface HoodieTableMetadata

Parameters:

partitionName - - Partition name

fileName - - File name for which bloom filter needs to be retrieved

Returns:

BloomFilter if available, otherwise empty

Throws:

HoodieMetadataException

getBloomFilters

public Map<Pair<String,String>,BloomFilter> getBloomFilters(List<Pair<String,String>> partitionNameFileNameList)
                                                     throws HoodieMetadataException

Description copied from interface: HoodieTableMetadata

Get bloom filters for files from the metadata table index.

Specified by:

getBloomFilters in interface HoodieTableMetadata

Parameters:

partitionNameFileNameList - - List of partition and file name pair for which bloom filters need to be retrieved

Returns:

Map of partition file name pair to its bloom filter

Throws:

HoodieMetadataException

getColumnStats

public Map<Pair<String,String>,HoodieMetadataColumnStats> getColumnStats(List<Pair<String,String>> partitionNameFileNameList,
                                                                         String columnName)
                                                                  throws HoodieMetadataException

Description copied from interface: HoodieTableMetadata

Get column stats for files from the metadata table index.

Specified by:

getColumnStats in interface HoodieTableMetadata

Parameters:

partitionNameFileNameList - - List of partition and file name pair for which bloom filters need to be retrieved

columnName - - Column name for which stats are needed

Returns:

Map of partition and file name pair to its column stats

Throws:

HoodieMetadataException

getRecordsByKeyPrefixes

public HoodieData<HoodieRecord<HoodieMetadataPayload>> getRecordsByKeyPrefixes(List<String> keyPrefixes,
                                                                               String partitionName,
                                                                               boolean shouldLoadInMemory)

Description copied from interface: HoodieTableMetadata

Fetch records by key prefixes. Key prefix passed is expected to match the same prefix as stored in Metadata table partitions. For eg, in case of col stats partition, actual keys in metadata partition is encoded values of column name, partition name and file name. So, key prefixes passed to this method is expected to be encoded already.

Specified by:

getRecordsByKeyPrefixes in interface HoodieTableMetadata

Parameters:

keyPrefixes - list of key prefixes for which interested records are looked up for.

partitionName - partition name in metadata table where the records are looked up for.

Returns:

HoodieData of HoodieRecords with records matching the passed in key prefixes.