org.apache.hudi

Class BaseHoodieTableFileIndex

java.lang.Object

org.apache.hudi.BaseHoodieTableFileIndex

All Implemented Interfaces:

AutoCloseable

public abstract class BaseHoodieTableFileIndex
extends Object
implements AutoCloseable

Common (engine-agnostic) File Index implementation enabling individual query engines to list Hudi Table contents based on the following:

• Table type (MOR, COW)
• Query type (snapshot, read_optimized, incremental)
• Query instant/range

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static interface	BaseHoodieTableFileIndex.FileStatusCache
static class	BaseHoodieTableFileIndex.PartitionPath

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.fs.Path	basePath
protected Map<BaseHoodieTableFileIndex.PartitionPath,List<FileSlice>>	cachedAllInputFileSlices
protected long	cachedFileSize
protected HoodieMetadataConfig	metadataConfig
protected boolean	queryAsNonePartitionedTable
protected List<org.apache.hadoop.fs.Path>	queryPaths

Constructor Summary

Constructors

Constructor and Description
BaseHoodieTableFileIndex(HoodieEngineContext engineContext, HoodieTableMetaClient metaClient, TypedProperties configProperties, HoodieTableQueryType queryType, List<org.apache.hadoop.fs.Path> queryPaths, Option<String> specifiedQueryInstant, boolean shouldIncludePendingCommits, boolean shouldValidateInstant, BaseHoodieTableFileIndex.FileStatusCache fileStatusCache)

Method Summary

Modifier and Type	Method and Description
void	close()
protected HoodieTimeline	getActiveTimeline()
protected List<BaseHoodieTableFileIndex.PartitionPath>	getAllQueryPartitionPaths()
String	getBasePath() Returns table's base-path
int	getFileSlicesCount()
Option<HoodieInstant>	getLatestCompletedInstant() Returns latest completed instant as seen by this instance of the file-index
Map<String,List<FileSlice>>	listFileSlices() Fetch list of latest base files and log files per partition.
protected abstract Object[]	parsePartitionColumnValues(String[] partitionColumns, String partitionPath)
protected void	refresh()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

metadataConfig

protected final HoodieMetadataConfig metadataConfig

queryPaths

protected final List<org.apache.hadoop.fs.Path> queryPaths

basePath

protected final org.apache.hadoop.fs.Path basePath

cachedFileSize

protected transient volatile long cachedFileSize

cachedAllInputFileSlices

protected transient volatile Map<BaseHoodieTableFileIndex.PartitionPath,List<FileSlice>> cachedAllInputFileSlices

queryAsNonePartitionedTable

protected volatile boolean queryAsNonePartitionedTable

Constructor Detail

BaseHoodieTableFileIndex

public BaseHoodieTableFileIndex(HoodieEngineContext engineContext,
                                HoodieTableMetaClient metaClient,
                                TypedProperties configProperties,
                                HoodieTableQueryType queryType,
                                List<org.apache.hadoop.fs.Path> queryPaths,
                                Option<String> specifiedQueryInstant,
                                boolean shouldIncludePendingCommits,
                                boolean shouldValidateInstant,
                                BaseHoodieTableFileIndex.FileStatusCache fileStatusCache)

Parameters:

engineContext - Hudi engine-specific context

metaClient - Hudi table's meta-client

configProperties - unifying configuration (in the form of generic properties)

queryType - target query type

queryPaths - target DFS paths being queried

specifiedQueryInstant - instant as of which table is being queried

shouldIncludePendingCommits - flags whether file-index should exclude any pending operations

shouldValidateInstant - flags to validate whether query instant is present in the timeline

fileStatusCache - transient cache of fetched [[FileStatus]]es

Method Detail

parsePartitionColumnValues

protected abstract Object[] parsePartitionColumnValues(String[] partitionColumns,
                                                       String partitionPath)

getLatestCompletedInstant

public Option<HoodieInstant> getLatestCompletedInstant()

Returns latest completed instant as seen by this instance of the file-index

getBasePath

public String getBasePath()

Returns table's base-path

listFileSlices

public Map<String,List<FileSlice>> listFileSlices()

Fetch list of latest base files and log files per partition.

Returns:

mapping from string partition paths to its base/log files

getFileSlicesCount

public int getFileSlicesCount()

close

public void close()
           throws Exception

Specified by:

close in interface AutoCloseable

Throws:

Exception

getAllQueryPartitionPaths

protected List<BaseHoodieTableFileIndex.PartitionPath> getAllQueryPartitionPaths()

refresh

protected void refresh()

getActiveTimeline

protected HoodieTimeline getActiveTimeline()