org.apache.hudi.metadata

Class JavaHoodieBackedTableMetadataWriter

java.lang.Object

org.apache.hudi.metadata.HoodieBackedTableMetadataWriter<List<org.apache.hudi.common.model.HoodieRecord>>

org.apache.hudi.metadata.JavaHoodieBackedTableMetadataWriter

All Implemented Interfaces:

Serializable, AutoCloseable, HoodieTableMetadataWriter

public class JavaHoodieBackedTableMetadataWriter
extends HoodieBackedTableMetadataWriter<List<org.apache.hudi.common.model.HoodieRecord>>

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.hudi.metadata.HoodieBackedTableMetadataWriter

dataMetaClient, dataWriteConfig, enabledPartitionTypes, engineContext, hadoopConf, metadata, METADATA_COMPACTION_TIME_SUFFIX, metadataMetaClient, metadataWriteConfig, metrics

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	JavaHoodieBackedTableMetadataWriter(org.apache.hadoop.conf.Configuration hadoopConf, HoodieWriteConfig writeConfig, org.apache.hudi.common.model.HoodieFailedWritesCleaningPolicy failedWritesCleaningPolicy, org.apache.hudi.common.engine.HoodieEngineContext engineContext, org.apache.hudi.common.util.Option<String> inflightInstantTimestamp) Hudi backed table metadata writer.

Method Summary

Modifier and Type	Method and Description
protected void	bulkCommit(String instantTime, MetadataPartitionType partitionType, org.apache.hudi.common.data.HoodieData<org.apache.hudi.common.model.HoodieRecord> records, int fileGroupCount)
protected void	commit(String instantTime, Map<MetadataPartitionType,org.apache.hudi.common.data.HoodieData<org.apache.hudi.common.model.HoodieRecord>> partitionRecordsMap)
protected List<org.apache.hudi.common.model.HoodieRecord>	convertHoodieDataToEngineSpecificData(org.apache.hudi.common.data.HoodieData<org.apache.hudi.common.model.HoodieRecord> records)
static HoodieTableMetadataWriter	create(org.apache.hadoop.conf.Configuration conf, HoodieWriteConfig writeConfig, org.apache.hudi.common.engine.HoodieEngineContext context, org.apache.hudi.common.util.Option<String> inflightInstantTimestamp)
static HoodieTableMetadataWriter	create(org.apache.hadoop.conf.Configuration conf, HoodieWriteConfig writeConfig, org.apache.hudi.common.model.HoodieFailedWritesCleaningPolicy failedWritesCleaningPolicy, org.apache.hudi.common.engine.HoodieEngineContext context, org.apache.hudi.common.util.Option<String> inflightInstantTimestamp)
void	deletePartitions(String instantTime, List<MetadataPartitionType> partitions)
protected org.apache.hudi.common.data.HoodieData<org.apache.hudi.common.model.HoodieRecord>	getFunctionalIndexRecords(List<org.apache.hudi.common.util.collection.Pair<String,org.apache.hudi.common.model.FileSlice>> partitionFileSlicePairs, org.apache.hudi.common.model.HoodieFunctionalIndexDefinition indexDefinition, org.apache.hudi.common.table.HoodieTableMetaClient metaClient, int parallelism, org.apache.avro.Schema readerSchema, org.apache.hudi.common.config.SerializableConfiguration hadoopConf)
protected BaseHoodieWriteClient<?,List<org.apache.hudi.common.model.HoodieRecord>,?,?>	initializeWriteClient()
protected void	initRegistry()
protected void	preWrite(String instantTime)

Methods inherited from class org.apache.hudi.metadata.HoodieBackedTableMetadataWriter

buildMetadataPartitions, checkNumDeltaCommits, cleanIfNecessary, close, closeInternal, commitInternal, compactIfNecessary, dropMetadataPartitions, getEnabledPartitionTypes, getTableMetadata, getWriteClient, getWriteConfig, initializeIfNeeded, isInitialized, performTableServices, prepRecords, update, update, update, update, updateFromWriteStatuses, validateRollback, validateTimelineBeforeSchedulingCompaction

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

JavaHoodieBackedTableMetadataWriter

protected JavaHoodieBackedTableMetadataWriter(org.apache.hadoop.conf.Configuration hadoopConf,
                                              HoodieWriteConfig writeConfig,
                                              org.apache.hudi.common.model.HoodieFailedWritesCleaningPolicy failedWritesCleaningPolicy,
                                              org.apache.hudi.common.engine.HoodieEngineContext engineContext,
                                              org.apache.hudi.common.util.Option<String> inflightInstantTimestamp)

Hudi backed table metadata writer.

Parameters:

hadoopConf - Hadoop configuration to use for the metadata writer

writeConfig - Writer config

failedWritesCleaningPolicy - Cleaning policy on failed writes

engineContext - Engine context

inflightInstantTimestamp - Timestamp of any instant in progress

Method Detail

create

public static HoodieTableMetadataWriter create(org.apache.hadoop.conf.Configuration conf,
                                               HoodieWriteConfig writeConfig,
                                               org.apache.hudi.common.engine.HoodieEngineContext context,
                                               org.apache.hudi.common.util.Option<String> inflightInstantTimestamp)

create

public static HoodieTableMetadataWriter create(org.apache.hadoop.conf.Configuration conf,
                                               HoodieWriteConfig writeConfig,
                                               org.apache.hudi.common.model.HoodieFailedWritesCleaningPolicy failedWritesCleaningPolicy,
                                               org.apache.hudi.common.engine.HoodieEngineContext context,
                                               org.apache.hudi.common.util.Option<String> inflightInstantTimestamp)

initRegistry

protected void initRegistry()

Specified by:

initRegistry in class HoodieBackedTableMetadataWriter<List<org.apache.hudi.common.model.HoodieRecord>>

commit

protected void commit(String instantTime,
                      Map<MetadataPartitionType,org.apache.hudi.common.data.HoodieData<org.apache.hudi.common.model.HoodieRecord>> partitionRecordsMap)

Specified by:

commit in class HoodieBackedTableMetadataWriter<List<org.apache.hudi.common.model.HoodieRecord>>

convertHoodieDataToEngineSpecificData

protected List<org.apache.hudi.common.model.HoodieRecord> convertHoodieDataToEngineSpecificData(org.apache.hudi.common.data.HoodieData<org.apache.hudi.common.model.HoodieRecord> records)

Specified by:

convertHoodieDataToEngineSpecificData in class HoodieBackedTableMetadataWriter<List<org.apache.hudi.common.model.HoodieRecord>>

bulkCommit

protected void bulkCommit(String instantTime,
                          MetadataPartitionType partitionType,
                          org.apache.hudi.common.data.HoodieData<org.apache.hudi.common.model.HoodieRecord> records,
                          int fileGroupCount)

Specified by:

bulkCommit in class HoodieBackedTableMetadataWriter<List<org.apache.hudi.common.model.HoodieRecord>>

initializeWriteClient

protected BaseHoodieWriteClient<?,List<org.apache.hudi.common.model.HoodieRecord>,?,?> initializeWriteClient()

Specified by:

initializeWriteClient in class HoodieBackedTableMetadataWriter<List<org.apache.hudi.common.model.HoodieRecord>>

deletePartitions

public void deletePartitions(String instantTime,
                             List<MetadataPartitionType> partitions)

preWrite

protected void preWrite(String instantTime)

Overrides:

preWrite in class HoodieBackedTableMetadataWriter<List<org.apache.hudi.common.model.HoodieRecord>>

getFunctionalIndexRecords

protected org.apache.hudi.common.data.HoodieData<org.apache.hudi.common.model.HoodieRecord> getFunctionalIndexRecords(List<org.apache.hudi.common.util.collection.Pair<String,org.apache.hudi.common.model.FileSlice>> partitionFileSlicePairs,
                                                                                                                      org.apache.hudi.common.model.HoodieFunctionalIndexDefinition indexDefinition,
                                                                                                                      org.apache.hudi.common.table.HoodieTableMetaClient metaClient,
                                                                                                                      int parallelism,
                                                                                                                      org.apache.avro.Schema readerSchema,
                                                                                                                      org.apache.hudi.common.config.SerializableConfiguration hadoopConf)

Specified by:

getFunctionalIndexRecords in class HoodieBackedTableMetadataWriter<List<org.apache.hudi.common.model.HoodieRecord>>