org.apache.hudi.common.table

Class HoodieTableMetaClient

java.lang.Object

org.apache.hudi.common.table.HoodieTableMetaClient

All Implemented Interfaces:

Serializable

public class HoodieTableMetaClient
extends Object
implements Serializable

HoodieTableMetaClient allows to access meta-data about a hoodie table It returns meta-data about commits, savepoints, compactions, cleanups as a HoodieTimeline Create an instance of the HoodieTableMetaClient with FileSystem and basePath to start getting the meta-data.

All the timelines are computed lazily, once computed the timeline is cached and never refreshed. Use the HoodieTimeline.reload() to refresh timelines.

Since:

0.3.0

See Also:

HoodieTimeline, Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HoodieTableMetaClient.Builder Builder for HoodieTableMetaClient.
static class	HoodieTableMetaClient.PropertyBuilder

Field Summary

Fields

Modifier and Type	Field and Description
protected HoodieActiveTimeline	activeTimeline
static String	AUXILIARYFOLDER_NAME
protected SerializablePath	basePath
static String	BOOTSTRAP_INDEX_BY_FILE_ID_FOLDER_PATH
static String	BOOTSTRAP_INDEX_BY_PARTITION_FOLDER_PATH
static String	BOOTSTRAP_INDEX_ROOT_FOLDER_PATH
protected SerializableConfiguration	hadoopConf
static String	HASHING_METADATA_FOLDER_NAME
static String	HEARTBEAT_FOLDER_NAME
static String	MARKER_EXTN
static String	METADATA_TABLE_FOLDER_PATH
static String	METAFOLDER_NAME
protected SerializablePath	metaPath
protected HoodieMetastoreConfig	metastoreConfig
static String	SCHEMA_FOLDER_NAME
protected HoodieTableConfig	tableConfig
static String	TEMPFOLDER_NAME

Constructor Summary

Constructors

Modifier	Constructor and Description
	HoodieTableMetaClient() Deprecated.
protected	HoodieTableMetaClient(org.apache.hadoop.conf.Configuration conf, String basePath, boolean loadActiveTimelineOnLoad, ConsistencyGuardConfig consistencyGuardConfig, Option<TimelineLayoutVersion> layoutVersion, String payloadClassName, FileSystemRetryConfig fileSystemRetryConfig)

Method Summary

Modifier and Type	Method and Description
static HoodieTableMetaClient.Builder	builder()
boolean	equals(Object o)
HoodieActiveTimeline	getActiveTimeline() Get the active instants as a timeline.
HoodieArchivedTimeline	getArchivedTimeline() Get the archived commits as a timeline.
HoodieArchivedTimeline	getArchivedTimeline(String startTs) Returns fresh new archived commits as a timeline from startTs (inclusive).
String	getArchivePath()
String	getBasePath() Deprecated. please use getBasePathV2()
org.apache.hadoop.fs.Path	getBasePathV2() Returns base path of the table
String	getBootstrapIndexByFileIdFolderNameFolderPath()
String	getBootstrapIndexByPartitionFolderPath()
String	getCommitActionType() Gets the commit action type.
HoodieTimeline	getCommitsAndCompactionTimeline() Get the commit + pending-compaction timeline visible for this table.
HoodieTimeline	getCommitsTimeline() Get the commit timeline visible for this table.
HoodieTimeline	getCommitTimeline() Get the compacted commit timeline visible for this table.
ConsistencyGuardConfig	getConsistencyGuardConfig()
FileSystemRetryConfig	getFileSystemRetryConfig()
HoodieWrapperFileSystem	getFs() Get the FS implementation for this table.
org.apache.hadoop.conf.Configuration	getHadoopConf()
String	getHashingMetadataPath()
static String	getHeartbeatFolderPath(String basePath)
String	getMarkerFolderPath(String instantTs) Returns Marker folder path.
String	getMetaAuxiliaryPath()
String	getMetaPath()
HoodieMetastoreConfig	getMetastoreConfig()
org.apache.hadoop.fs.FileSystem	getRawFs() Return raw file-system.
String	getSchemaFolderName()
HoodieTableConfig	getTableConfig()
HoodieTableType	getTableType()
String	getTempFolderPath()
TimelineLayoutVersion	getTimelineLayoutVersion()
int	hashCode()
void	initializeBootstrapDirsIfNotExists()
static void	initializeBootstrapDirsIfNotExists(org.apache.hadoop.conf.Configuration hadoopConf, String basePath, org.apache.hadoop.fs.FileSystem fs)
static HoodieTableMetaClient	initTableAndGetMetaClient(org.apache.hadoop.conf.Configuration hadoopConf, String basePath, Properties props) Helper method to initialize a given path as a hoodie table with configs passed in as Properties.
boolean	isTimelineNonEmpty()
static HoodieTableMetaClient	reload(HoodieTableMetaClient oldMetaClient)
HoodieActiveTimeline	reloadActiveTimeline() Reload ActiveTimeline and cache.
static org.apache.hadoop.fs.FileStatus[]	scanFiles(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path metaPath, org.apache.hadoop.fs.PathFilter nameFilter) Helper method to scan all hoodie-instant metafiles.
List<HoodieInstant>	scanHoodieInstantsFromFileSystem(org.apache.hadoop.fs.Path timelinePath, Set<String> includedExtensions, boolean applyLayoutVersionFilters) Helper method to scan all hoodie-instant metafiles and construct HoodieInstant objects.
List<HoodieInstant>	scanHoodieInstantsFromFileSystem(Set<String> includedExtensions, boolean applyLayoutVersionFilters) Helper method to scan all hoodie-instant metafiles and construct HoodieInstant objects.
void	setFs(HoodieWrapperFileSystem fs)
String	toString()
void	validateTableProperties(Properties properties) Validate table properties.
static HoodieTableMetaClient.PropertyBuilder	withPropertyBuilder()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

METAFOLDER_NAME

public static final String METAFOLDER_NAME

See Also:

Constant Field Values

TEMPFOLDER_NAME

public static final String TEMPFOLDER_NAME

See Also:

Constant Field Values

AUXILIARYFOLDER_NAME

public static final String AUXILIARYFOLDER_NAME

See Also:

Constant Field Values

BOOTSTRAP_INDEX_ROOT_FOLDER_PATH

public static final String BOOTSTRAP_INDEX_ROOT_FOLDER_PATH

See Also:

Constant Field Values

HEARTBEAT_FOLDER_NAME

public static final String HEARTBEAT_FOLDER_NAME

See Also:

Constant Field Values

METADATA_TABLE_FOLDER_PATH

public static final String METADATA_TABLE_FOLDER_PATH

See Also:

Constant Field Values

HASHING_METADATA_FOLDER_NAME

public static final String HASHING_METADATA_FOLDER_NAME

See Also:

Constant Field Values

BOOTSTRAP_INDEX_BY_PARTITION_FOLDER_PATH

public static final String BOOTSTRAP_INDEX_BY_PARTITION_FOLDER_PATH

See Also:

Constant Field Values

BOOTSTRAP_INDEX_BY_FILE_ID_FOLDER_PATH

public static final String BOOTSTRAP_INDEX_BY_FILE_ID_FOLDER_PATH

See Also:

Constant Field Values

SCHEMA_FOLDER_NAME

public static final String SCHEMA_FOLDER_NAME

See Also:

Constant Field Values

MARKER_EXTN

public static final String MARKER_EXTN

See Also:

Constant Field Values

basePath

protected SerializablePath basePath

metaPath

protected SerializablePath metaPath

hadoopConf

protected SerializableConfiguration hadoopConf

tableConfig

protected HoodieTableConfig tableConfig

activeTimeline

protected HoodieActiveTimeline activeTimeline

metastoreConfig

protected HoodieMetastoreConfig metastoreConfig

Constructor Detail

HoodieTableMetaClient

protected HoodieTableMetaClient(org.apache.hadoop.conf.Configuration conf,
                                String basePath,
                                boolean loadActiveTimelineOnLoad,
                                ConsistencyGuardConfig consistencyGuardConfig,
                                Option<TimelineLayoutVersion> layoutVersion,
                                String payloadClassName,
                                FileSystemRetryConfig fileSystemRetryConfig)

HoodieTableMetaClient

public HoodieTableMetaClient()

Deprecated.

For serializing and de-serializing.

Method Detail

reload

public static HoodieTableMetaClient reload(HoodieTableMetaClient oldMetaClient)

getBasePathV2

public org.apache.hadoop.fs.Path getBasePathV2()

Returns base path of the table

getBasePath

@Deprecated
public String getBasePath()

Deprecated. please use getBasePathV2()

Returns:

Base path

getTableType

public HoodieTableType getTableType()

Returns:

Hoodie Table Type

getMetaPath

public String getMetaPath()

Returns:

Meta path

getSchemaFolderName

public String getSchemaFolderName()

Returns:

schema folder path

getHashingMetadataPath

public String getHashingMetadataPath()

Returns:

Hashing metadata base path

getTempFolderPath

public String getTempFolderPath()

Returns:

Temp Folder path

getMarkerFolderPath

public String getMarkerFolderPath(String instantTs)

Returns Marker folder path.

Parameters:

instantTs - Instant Timestamp

Returns:

getMetaAuxiliaryPath

public String getMetaAuxiliaryPath()

Returns:

Auxiliary Meta path

getHeartbeatFolderPath

public static String getHeartbeatFolderPath(String basePath)

Returns:

Heartbeat folder path.

getBootstrapIndexByPartitionFolderPath

public String getBootstrapIndexByPartitionFolderPath()

Returns:

Bootstrap Index By Partition Folder

getBootstrapIndexByFileIdFolderNameFolderPath

public String getBootstrapIndexByFileIdFolderNameFolderPath()

Returns:

Bootstrap Index By Hudi File Id Folder

getArchivePath

public String getArchivePath()

Returns:

path where archived timeline is stored

getTableConfig

public HoodieTableConfig getTableConfig()

Returns:

Table Config

getTimelineLayoutVersion

public TimelineLayoutVersion getTimelineLayoutVersion()

getFs

public HoodieWrapperFileSystem getFs()

Get the FS implementation for this table.

setFs

public void setFs(HoodieWrapperFileSystem fs)

getRawFs

public org.apache.hadoop.fs.FileSystem getRawFs()

Return raw file-system.

Returns:

fs

getHadoopConf

public org.apache.hadoop.conf.Configuration getHadoopConf()

getActiveTimeline

public HoodieActiveTimeline getActiveTimeline()

Get the active instants as a timeline.

Returns:

Active instants timeline

reloadActiveTimeline

public HoodieActiveTimeline reloadActiveTimeline()

Reload ActiveTimeline and cache.

Returns:

Active instants timeline

getConsistencyGuardConfig

public ConsistencyGuardConfig getConsistencyGuardConfig()

getFileSystemRetryConfig

public FileSystemRetryConfig getFileSystemRetryConfig()

getArchivedTimeline

public HoodieArchivedTimeline getArchivedTimeline()

Get the archived commits as a timeline. This is costly operation, as all data from the archived files are read. This should not be used, unless for historical debugging purposes.

Returns:

Archived commit timeline

getMetastoreConfig

public HoodieMetastoreConfig getMetastoreConfig()

getArchivedTimeline

public HoodieArchivedTimeline getArchivedTimeline(String startTs)

Returns fresh new archived commits as a timeline from startTs (inclusive).

This is costly operation if really early endTs is specified. Be caution to use this only when the time range is short.

This method is not thread safe.

Returns:

Archived commit timeline

validateTableProperties

public void validateTableProperties(Properties properties)

Validate table properties.

Parameters:

properties - Properties from writeConfig.

initTableAndGetMetaClient

public static HoodieTableMetaClient initTableAndGetMetaClient(org.apache.hadoop.conf.Configuration hadoopConf,
                                                              String basePath,
                                                              Properties props)
                                                       throws IOException

Helper method to initialize a given path as a hoodie table with configs passed in as Properties.

Returns:

Instance of HoodieTableMetaClient

Throws:

IOException

initializeBootstrapDirsIfNotExists

public static void initializeBootstrapDirsIfNotExists(org.apache.hadoop.conf.Configuration hadoopConf,
                                                      String basePath,
                                                      org.apache.hadoop.fs.FileSystem fs)
                                               throws IOException

Throws:

IOException

scanFiles

public static org.apache.hadoop.fs.FileStatus[] scanFiles(org.apache.hadoop.fs.FileSystem fs,
                                                          org.apache.hadoop.fs.Path metaPath,
                                                          org.apache.hadoop.fs.PathFilter nameFilter)
                                                   throws IOException

Helper method to scan all hoodie-instant metafiles.

Parameters:

fs - The file system implementation for this table

metaPath - The meta path where meta files are stored

nameFilter - The name filter to filter meta files

Returns:

An array of meta FileStatus

Throws:

IOException - In case of failure

isTimelineNonEmpty

public boolean isTimelineNonEmpty()

Returns:

true if any commits are found, else false.

getCommitsTimeline

public HoodieTimeline getCommitsTimeline()

Get the commit timeline visible for this table.

getCommitsAndCompactionTimeline

public HoodieTimeline getCommitsAndCompactionTimeline()

Get the commit + pending-compaction timeline visible for this table. A RT filesystem view is constructed with this timeline so that file-slice after pending compaction-requested instant-time is also considered valid. A RT file-system view for reading must then merge the file-slices before and after pending compaction instant so that all delta-commits are read.

getCommitTimeline

public HoodieTimeline getCommitTimeline()

Get the compacted commit timeline visible for this table.

getCommitActionType

public String getCommitActionType()

Gets the commit action type.

scanHoodieInstantsFromFileSystem

public List<HoodieInstant> scanHoodieInstantsFromFileSystem(Set<String> includedExtensions,
                                                            boolean applyLayoutVersionFilters)
                                                     throws IOException

Helper method to scan all hoodie-instant metafiles and construct HoodieInstant objects.

Parameters:

includedExtensions - Included hoodie extensions

applyLayoutVersionFilters - Depending on Timeline layout version, if there are multiple states for the same action instant, only include the highest state

Returns:

List of Hoodie Instants generated

Throws:

IOException - in case of failure

scanHoodieInstantsFromFileSystem

public List<HoodieInstant> scanHoodieInstantsFromFileSystem(org.apache.hadoop.fs.Path timelinePath,
                                                            Set<String> includedExtensions,
                                                            boolean applyLayoutVersionFilters)
                                                     throws IOException

Helper method to scan all hoodie-instant metafiles and construct HoodieInstant objects.

Parameters:

timelinePath - MetaPath where instant files are stored

includedExtensions - Included hoodie extensions

applyLayoutVersionFilters - Depending on Timeline layout version, if there are multiple states for the same action instant, only include the highest state

Returns:

List of Hoodie Instants generated

Throws:

IOException - in case of failure

equals

public boolean equals(Object o)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object

toString

public String toString()

Overrides:

toString in class Object

initializeBootstrapDirsIfNotExists

public void initializeBootstrapDirsIfNotExists()
                                        throws IOException

Throws:

IOException

builder

public static HoodieTableMetaClient.Builder builder()

withPropertyBuilder

public static HoodieTableMetaClient.PropertyBuilder withPropertyBuilder()