org.apache.hudi

Class BaseHoodieTableFileIndex

java.lang.Object

org.apache.hudi.BaseHoodieTableFileIndex

All Implemented Interfaces:

AutoCloseable

public abstract class BaseHoodieTableFileIndex
extends Object
implements AutoCloseable

Common (engine-agnostic) File Index implementation enabling individual query engines to list Hudi Table contents based on the

• Table type (MOR, COW)
• Query type (snapshot, read_optimized, incremental)
• Query instant/range

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static interface	BaseHoodieTableFileIndex.FileStatusCache APIs for caching StoragePathInfo.
static class	BaseHoodieTableFileIndex.PartitionPath Partition path information containing the relative partition path and values of partition columns.

Field Summary

Fields

Modifier and Type	Field and Description
protected HoodieMetadataConfig	metadataConfig

Constructor Summary

Constructors

Constructor and Description

BaseHoodieTableFileIndex(HoodieEngineContext engineContext, HoodieTableMetaClient metaClient, TypedProperties configProperties, HoodieTableQueryType queryType, List<StoragePath> queryPaths, Option<String> specifiedQueryInstant, boolean shouldIncludePendingCommits, boolean shouldValidateInstant, BaseHoodieTableFileIndex.FileStatusCache fileStatusCache, boolean shouldListLazily, Option<String> beginInstantTime, Option<String> endInstantTime)

Method Summary

Modifier and Type	Method and Description
protected boolean	areAllFileSlicesCached()
protected boolean	areAllPartitionPathsCached()
void	close()
protected abstract Object[]	doParsePartitionColumnValues(String[] partitionColumns, String partitionPath)
protected HoodieTimeline	getActiveTimeline()
protected Map<BaseHoodieTableFileIndex.PartitionPath,List<FileSlice>>	getAllInputFileSlices() Returns all listed file-slices w/in the partition paths returned by getAllQueryPartitionPaths()
protected List<BaseHoodieTableFileIndex.PartitionPath>	getAllQueryPartitionPaths() Returns all partition paths matching the ones explicitly provided by the query (if any)
StoragePath	getBasePath() Returns table's base-path
int	getFileSlicesCount()
protected Map<BaseHoodieTableFileIndex.PartitionPath,List<FileSlice>>	getInputFileSlices(BaseHoodieTableFileIndex.PartitionPath... partitions) Get input file slice for the given partition.
Option<HoodieInstant>	getLatestCompletedInstant() Returns latest completed instant as seen by this instance of the file-index
protected String[]	getPartitionColumns()
protected List<StoragePath>	getQueryPaths()
protected long	getTotalCachedFilesSize()
protected List<BaseHoodieTableFileIndex.PartitionPath>	listPartitionPaths(List<String> relativePartitionPaths)
protected List<BaseHoodieTableFileIndex.PartitionPath>	listPartitionPaths(List<String> relativePartitionPaths, Types.RecordType partitionFields, Expression partitionColumnPredicates)
protected void	refresh()
protected boolean	shouldReadAsPartitionedTable()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

metadataConfig

protected final HoodieMetadataConfig metadataConfig

Constructor Detail

BaseHoodieTableFileIndex

public BaseHoodieTableFileIndex(HoodieEngineContext engineContext,
                                HoodieTableMetaClient metaClient,
                                TypedProperties configProperties,
                                HoodieTableQueryType queryType,
                                List<StoragePath> queryPaths,
                                Option<String> specifiedQueryInstant,
                                boolean shouldIncludePendingCommits,
                                boolean shouldValidateInstant,
                                BaseHoodieTableFileIndex.FileStatusCache fileStatusCache,
                                boolean shouldListLazily,
                                Option<String> beginInstantTime,
                                Option<String> endInstantTime)

Parameters:

engineContext - Hudi engine-specific context

metaClient - Hudi table's meta-client

configProperties - unifying configuration (in the form of generic properties)

queryType - target query type

queryPaths - target DFS paths being queried

specifiedQueryInstant - instant as of which table is being queried

shouldIncludePendingCommits - flags whether file-index should exclude any pending operations

shouldValidateInstant - flags to validate whether query instant is present in the timeline

fileStatusCache - transient cache of fetched [[FileStatus]]es

beginInstantTime - begin instant time for incremental query (optional)

endInstantTime - end instant time for incremental query (optional)

Method Detail

doParsePartitionColumnValues

protected abstract Object[] doParsePartitionColumnValues(String[] partitionColumns,
                                                         String partitionPath)

getLatestCompletedInstant

public Option<HoodieInstant> getLatestCompletedInstant()

Returns latest completed instant as seen by this instance of the file-index

getBasePath

public StoragePath getBasePath()

Returns table's base-path

getFileSlicesCount

public int getFileSlicesCount()

close

public void close()
           throws Exception

Specified by:

close in interface AutoCloseable

Throws:

Exception

getPartitionColumns

protected String[] getPartitionColumns()

getQueryPaths

protected List<StoragePath> getQueryPaths()

getAllQueryPartitionPaths

protected List<BaseHoodieTableFileIndex.PartitionPath> getAllQueryPartitionPaths()

Returns all partition paths matching the ones explicitly provided by the query (if any)

getAllInputFileSlices

protected Map<BaseHoodieTableFileIndex.PartitionPath,List<FileSlice>> getAllInputFileSlices()

Returns all listed file-slices w/in the partition paths returned by getAllQueryPartitionPaths()

getInputFileSlices

protected Map<BaseHoodieTableFileIndex.PartitionPath,List<FileSlice>> getInputFileSlices(BaseHoodieTableFileIndex.PartitionPath... partitions)

Get input file slice for the given partition. Will use cache directly if it is computed before.

listPartitionPaths

protected List<BaseHoodieTableFileIndex.PartitionPath> listPartitionPaths(List<String> relativePartitionPaths,
                                                                          Types.RecordType partitionFields,
                                                                          Expression partitionColumnPredicates)

listPartitionPaths

protected List<BaseHoodieTableFileIndex.PartitionPath> listPartitionPaths(List<String> relativePartitionPaths)

refresh

protected void refresh()

getActiveTimeline

protected HoodieTimeline getActiveTimeline()

getTotalCachedFilesSize

protected long getTotalCachedFilesSize()

areAllFileSlicesCached

protected boolean areAllFileSlicesCached()

areAllPartitionPathsCached

protected boolean areAllPartitionPathsCached()

shouldReadAsPartitionedTable

protected boolean shouldReadAsPartitionedTable()