org.apache.hudi.metadata

Class HoodieBackedTableMetadataWriter<I>

java.lang.Object

org.apache.hudi.metadata.HoodieBackedTableMetadataWriter<I>

Type Parameters:

I - Type of input for the write client

All Implemented Interfaces:

Serializable, AutoCloseable, HoodieTableMetadataWriter

public abstract class HoodieBackedTableMetadataWriter<I>
extends Object
implements HoodieTableMetadataWriter

Writer implementation backed by an internal hudi table. Partition and file listing are saved within an internal MOR table called Metadata Table. This table is created by listing files and partitions (first time) and kept in sync using the instants on the main dataset.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected HoodieTableMetaClient	dataMetaClient
protected HoodieWriteConfig	dataWriteConfig
protected List<MetadataPartitionType>	enabledPartitionTypes
protected HoodieEngineContext	engineContext
protected SerializableConfiguration	hadoopConf
protected HoodieBackedTableMetadata	metadata
static String	METADATA_COMPACTION_TIME_SUFFIX
protected HoodieTableMetaClient	metadataMetaClient
protected HoodieWriteConfig	metadataWriteConfig
protected Option<HoodieMetadataMetrics>	metrics

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	HoodieBackedTableMetadataWriter(org.apache.hadoop.conf.Configuration hadoopConf, HoodieWriteConfig writeConfig, HoodieFailedWritesCleaningPolicy failedWritesCleaningPolicy, HoodieEngineContext engineContext, Option<String> inflightInstantTimestamp) Hudi backed table metadata writer.

Method Summary

Modifier and Type	Method and Description
void	buildMetadataPartitions(HoodieEngineContext engineContext, List<HoodieIndexPartitionInfo> indexPartitionInfos) Builds the given metadata partitions to create index.
protected abstract void	bulkCommit(String instantTime, MetadataPartitionType partitionType, HoodieData<HoodieRecord> records, int fileGroupCount) Commit the HoodieRecords to Metadata Table as a new delta-commit using bulk commit (if supported).
protected static void	checkNumDeltaCommits(HoodieTableMetaClient metaClient, int maxNumDeltaCommitsWhenPending)
protected void	cleanIfNecessary(BaseHoodieWriteClient writeClient, String instantTime)
void	close()
protected void	closeInternal()
protected abstract void	commit(String instantTime, Map<MetadataPartitionType,HoodieData<HoodieRecord>> partitionRecordsMap) Commit the HoodieRecords to Metadata Table as a new delta-commit.
protected void	commitInternal(String instantTime, Map<MetadataPartitionType,HoodieData<HoodieRecord>> partitionRecordsMap, boolean isInitializing, Option<BulkInsertPartitioner> bulkInsertPartitioner)
protected void	compactIfNecessary(BaseHoodieWriteClient writeClient, String latestDeltacommitTime) Perform a compaction on the Metadata Table.
protected abstract I	convertHoodieDataToEngineSpecificData(HoodieData<HoodieRecord> records) Converts the input records to the input format expected by the write client.
void	dropMetadataPartitions(List<MetadataPartitionType> metadataPartitions) Drop the given metadata partitions.
List<MetadataPartitionType>	getEnabledPartitionTypes()
HoodieBackedTableMetadata	getTableMetadata()
protected BaseHoodieWriteClient<?,I,?,?>	getWriteClient()
HoodieWriteConfig	getWriteConfig()
protected boolean	initializeIfNeeded(HoodieTableMetaClient dataMetaClient, Option<String> inflightInstantTimestamp) Initialize the metadata table if needed.
protected abstract BaseHoodieWriteClient<?,I,?,?>	initializeWriteClient()
protected abstract void	initRegistry()
boolean	isInitialized() Returns true if the metadata table is initialized.
void	performTableServices(Option<String> inFlightInstantTimestamp) Optimize the metadata table by running compaction, clean and archive as required.
protected HoodieData<HoodieRecord>	prepRecords(Map<MetadataPartitionType,HoodieData<HoodieRecord>> partitionRecordsMap) Tag each record with the location in the given partition.
protected void	preWrite(String instantTime) Allows the implementation to perform any pre-commit operations like transitioning a commit to inflight if required.
void	update(HoodieCleanMetadata cleanMetadata, String instantTime) Update from HoodieCleanMetadata.
void	update(HoodieCommitMetadata commitMetadata, HoodieData<WriteStatus> writeStatus, String instantTime) Update from HoodieCommitMetadata.
void	update(HoodieRestoreMetadata restoreMetadata, String instantTime) Update from HoodieRestoreMetadata.
void	update(HoodieRollbackMetadata rollbackMetadata, String instantTime) Update from HoodieRollbackMetadata.
protected void	validateRollback(String commitToRollbackInstantTime, HoodieInstant compactionInstant, HoodieTimeline deltacommitsSinceCompaction)
protected boolean	validateTimelineBeforeSchedulingCompaction(Option<String> inFlightInstantTimestamp, String latestDeltaCommitTimeInMetadataTable) Validates the timeline for both main and metadata tables to ensure compaction on MDT can be scheduled.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hudi.metadata.HoodieTableMetadataWriter

deletePartitions

Field Detail

METADATA_COMPACTION_TIME_SUFFIX

public static final String METADATA_COMPACTION_TIME_SUFFIX

See Also:

Constant Field Values

metadataWriteConfig

protected HoodieWriteConfig metadataWriteConfig

dataWriteConfig

protected HoodieWriteConfig dataWriteConfig

metadata

protected HoodieBackedTableMetadata metadata

metadataMetaClient

protected HoodieTableMetaClient metadataMetaClient

dataMetaClient

protected HoodieTableMetaClient dataMetaClient

metrics

protected Option<HoodieMetadataMetrics> metrics

hadoopConf

protected SerializableConfiguration hadoopConf

engineContext

protected final transient HoodieEngineContext engineContext

enabledPartitionTypes

protected final List<MetadataPartitionType> enabledPartitionTypes

Constructor Detail

HoodieBackedTableMetadataWriter

protected HoodieBackedTableMetadataWriter(org.apache.hadoop.conf.Configuration hadoopConf,
                                          HoodieWriteConfig writeConfig,
                                          HoodieFailedWritesCleaningPolicy failedWritesCleaningPolicy,
                                          HoodieEngineContext engineContext,
                                          Option<String> inflightInstantTimestamp)

Hudi backed table metadata writer.

Parameters:

hadoopConf - Hadoop configuration to use for the metadata writer

writeConfig - Writer config

failedWritesCleaningPolicy - Cleaning policy on failed writes

engineContext - Engine context

inflightInstantTimestamp - Timestamp of any instant in progress

Method Detail

initRegistry

protected abstract void initRegistry()

getWriteConfig

public HoodieWriteConfig getWriteConfig()

getTableMetadata

public HoodieBackedTableMetadata getTableMetadata()

getEnabledPartitionTypes

public List<MetadataPartitionType> getEnabledPartitionTypes()

initializeIfNeeded

protected boolean initializeIfNeeded(HoodieTableMetaClient dataMetaClient,
                                     Option<String> inflightInstantTimestamp)
                              throws IOException

Initialize the metadata table if needed.

Parameters:

dataMetaClient - - meta client for the data table

inflightInstantTimestamp - - timestamp of an instant in progress on the dataset

Throws:

IOException - on errors

dropMetadataPartitions

public void dropMetadataPartitions(List<MetadataPartitionType> metadataPartitions)
                            throws IOException

Description copied from interface: HoodieTableMetadataWriter

Drop the given metadata partitions.

Specified by:

dropMetadataPartitions in interface HoodieTableMetadataWriter

Parameters:

metadataPartitions - List of MDT partitions to drop

Throws:

IOException - on failures

checkNumDeltaCommits

protected static void checkNumDeltaCommits(HoodieTableMetaClient metaClient,
                                           int maxNumDeltaCommitsWhenPending)

buildMetadataPartitions

public void buildMetadataPartitions(HoodieEngineContext engineContext,
                                    List<HoodieIndexPartitionInfo> indexPartitionInfos)
                             throws IOException

Description copied from interface: HoodieTableMetadataWriter

Builds the given metadata partitions to create index.

Specified by:

buildMetadataPartitions in interface HoodieTableMetadataWriter

indexPartitionInfos - - information about partitions to build such as partition type and base instant time

Throws:

IOException

update

public void update(HoodieCommitMetadata commitMetadata,
                   HoodieData<WriteStatus> writeStatus,
                   String instantTime)

Update from HoodieCommitMetadata.

Specified by:

update in interface HoodieTableMetadataWriter

Parameters:

commitMetadata - HoodieCommitMetadata

instantTime - Timestamp at which the commit was performed

update

public void update(HoodieCleanMetadata cleanMetadata,
                   String instantTime)

Update from HoodieCleanMetadata.

Specified by:

update in interface HoodieTableMetadataWriter

Parameters:

cleanMetadata - HoodieCleanMetadata

instantTime - Timestamp at which the clean was completed

update

public void update(HoodieRestoreMetadata restoreMetadata,
                   String instantTime)

Update from HoodieRestoreMetadata.

Specified by:

update in interface HoodieTableMetadataWriter

Parameters:

restoreMetadata - HoodieRestoreMetadata

instantTime - Timestamp at which the restore was performed

update

public void update(HoodieRollbackMetadata rollbackMetadata,
                   String instantTime)

Update from HoodieRollbackMetadata.

Specified by:

update in interface HoodieTableMetadataWriter

Parameters:

rollbackMetadata - HoodieRollbackMetadata

instantTime - Timestamp at which the rollback was performed

validateRollback

protected void validateRollback(String commitToRollbackInstantTime,
                                HoodieInstant compactionInstant,
                                HoodieTimeline deltacommitsSinceCompaction)

close

public void close()
           throws Exception

Specified by:

close in interface AutoCloseable

Throws:

Exception

commit

protected abstract void commit(String instantTime,
                               Map<MetadataPartitionType,HoodieData<HoodieRecord>> partitionRecordsMap)

Commit the HoodieRecords to Metadata Table as a new delta-commit.

Parameters:

instantTime - - Action instant time for this commit

partitionRecordsMap - - Map of partition type to its records to commit

convertHoodieDataToEngineSpecificData

protected abstract I convertHoodieDataToEngineSpecificData(HoodieData<HoodieRecord> records)

Converts the input records to the input format expected by the write client.

Parameters:

records - records to be converted

Returns:

converted records

commitInternal

protected void commitInternal(String instantTime,
                              Map<MetadataPartitionType,HoodieData<HoodieRecord>> partitionRecordsMap,
                              boolean isInitializing,
                              Option<BulkInsertPartitioner> bulkInsertPartitioner)

preWrite

protected void preWrite(String instantTime)

Allows the implementation to perform any pre-commit operations like transitioning a commit to inflight if required.

Parameters:

instantTime - time of commit

bulkCommit

protected abstract void bulkCommit(String instantTime,
                                   MetadataPartitionType partitionType,
                                   HoodieData<HoodieRecord> records,
                                   int fileGroupCount)

Commit the HoodieRecords to Metadata Table as a new delta-commit using bulk commit (if supported).

This is used to optimize the initial commit to the MDT partition which may contains a large number of records and hence is more suited to bulkInsert for write performance.

Parameters:

instantTime - - Action instant time for this commit

partitionType - - The MDT partition to which records are to be committed

records - - records to be bulk inserted

fileGroupCount - - The maximum number of file groups to which the records will be written.

prepRecords

protected HoodieData<HoodieRecord> prepRecords(Map<MetadataPartitionType,HoodieData<HoodieRecord>> partitionRecordsMap)

Tag each record with the location in the given partition. The record is tagged with respective file slice's location based on its record key.

performTableServices

public void performTableServices(Option<String> inFlightInstantTimestamp)

Optimize the metadata table by running compaction, clean and archive as required.

Don't perform optimization if there are inflight operations on the dataset. This is for two reasons: - The compaction will contain the correct data as all failed operations have been rolled back. - Clean/compaction etc. will have the highest timestamp on the MDT and we won't be adding new operations with smaller timestamps to metadata table (makes for easier debugging)

This adds the limitations that long-running async operations (clustering, etc.) may cause delay in such MDT optimizations. We will relax this after MDT code has been hardened.

Specified by:

performTableServices in interface HoodieTableMetadataWriter

Parameters:

inFlightInstantTimestamp - Timestamp of an instant which is in-progress. This instant is ignored while deciding if optimizations can be performed.

compactIfNecessary

protected void compactIfNecessary(BaseHoodieWriteClient writeClient,
                                  String latestDeltacommitTime)

Perform a compaction on the Metadata Table.

Cases to be handled: 1. We cannot perform compaction if there are previous inflight operations on the dataset. This is because a compacted metadata base file at time Tx should represent all the actions on the dataset till time Tx.

2. In multi-writer scenario, a parallel operation with a greater instantTime may have completed creating a deltacommit.

cleanIfNecessary

protected void cleanIfNecessary(BaseHoodieWriteClient writeClient,
                                String instantTime)

validateTimelineBeforeSchedulingCompaction

protected boolean validateTimelineBeforeSchedulingCompaction(Option<String> inFlightInstantTimestamp,
                                                             String latestDeltaCommitTimeInMetadataTable)

Validates the timeline for both main and metadata tables to ensure compaction on MDT can be scheduled.

closeInternal

protected void closeInternal()

isInitialized

public boolean isInitialized()

Description copied from interface: HoodieTableMetadataWriter

Returns true if the metadata table is initialized.

Specified by:

isInitialized in interface HoodieTableMetadataWriter

getWriteClient

protected BaseHoodieWriteClient<?,I,?,?> getWriteClient()

initializeWriteClient

protected abstract BaseHoodieWriteClient<?,I,?,?> initializeWriteClient()