org.apache.hudi.common.util

Class CommitUtils

java.lang.Object

org.apache.hudi.common.util.CommitUtils

public class CommitUtils
extends Object

Helper class to generate commit metadata.

Constructor Summary

Constructors

Constructor and Description
CommitUtils()

Method Summary

Modifier and Type	Method and Description
static HoodieCommitMetadata	buildMetadata(List<HoodieWriteStat> writeStats, Map<String,List<String>> partitionToReplaceFileIds, Option<Map<String,String>> extraMetadata, WriteOperationType operationType, String schemaToStoreInCommit, String commitActionType)
static String	getCommitActionType(HoodieTableType tableType) Gets the commit action type for given table type.
static String	getCommitActionType(WriteOperationType operation, HoodieTableType tableType) Gets the commit action type for given write operation and table type.
static HashMap<String,String>	getFileIdWithoutSuffixAndRelativePaths(Map<String,List<HoodieWriteStat>> partitionToWriteStats)
static HashMap<String,String>	getFileIdWithoutSuffixAndRelativePathsFromSpecificRecord(Map<String,List<HoodieWriteStat>> partitionToWriteStats)
static Option<HoodieCommitMetadata>	getLatestCommitMetadataWithValidCheckpointInfo(HoodieTimeline timeline, String checkpointKey) Process previous commits metadata in the timeline to determine the checkpoint given a checkpoint key.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CommitUtils

public CommitUtils()

Method Detail

getCommitActionType

public static String getCommitActionType(WriteOperationType operation,
                                         HoodieTableType tableType)

Gets the commit action type for given write operation and table type. Use this API when commit action type can differ not only on the basis of table type but also write operation type. For example, INSERT_OVERWRITE/INSERT_OVERWRITE_TABLE operations have REPLACE commit action type.

getCommitActionType

public static String getCommitActionType(HoodieTableType tableType)

Gets the commit action type for given table type. Note: Use this API only when the commit action type is not dependent on the write operation type. See getCommitActionType(WriteOperationType, HoodieTableType) for more details.

buildMetadata

public static HoodieCommitMetadata buildMetadata(List<HoodieWriteStat> writeStats,
                                                 Map<String,List<String>> partitionToReplaceFileIds,
                                                 Option<Map<String,String>> extraMetadata,
                                                 WriteOperationType operationType,
                                                 String schemaToStoreInCommit,
                                                 String commitActionType)

getFileIdWithoutSuffixAndRelativePathsFromSpecificRecord

public static HashMap<String,String> getFileIdWithoutSuffixAndRelativePathsFromSpecificRecord(Map<String,List<HoodieWriteStat>> partitionToWriteStats)

getFileIdWithoutSuffixAndRelativePaths

public static HashMap<String,String> getFileIdWithoutSuffixAndRelativePaths(Map<String,List<HoodieWriteStat>> partitionToWriteStats)

getLatestCommitMetadataWithValidCheckpointInfo

public static Option<HoodieCommitMetadata> getLatestCommitMetadataWithValidCheckpointInfo(HoodieTimeline timeline,
                                                                                          String checkpointKey)

Process previous commits metadata in the timeline to determine the checkpoint given a checkpoint key. NOTE: This is very similar in intent to DeltaSync#getLatestCommitMetadataWithValidCheckpointInfo except that different deployment models (deltastreamer or spark structured streaming) could have different checkpoint keys.

Parameters:

timeline - completed commits in active timeline.

checkpointKey - the checkpoint key in the extra metadata of the commit.

Returns:

An optional commit metadata with latest checkpoint.