org.apache.hudi.common.util

Class CompactionUtils

java.lang.Object

org.apache.hudi.common.util.CompactionUtils

public class CompactionUtils
extends Object

Helper class to generate compaction plan from FileGroup/FileSlice abstraction.

Field Summary

Fields

Modifier and Type	Field and Description
static Integer	COMPACTION_METADATA_VERSION_1
static Integer	COMPACTION_METADATA_VERSION_2
static Integer	LATEST_COMPACTION_METADATA_VERSION

Constructor Summary

Constructors

Constructor and Description
CompactionUtils()

Method Summary

Modifier and Type	Method and Description
static CompactionOperation	buildCompactionOperation(HoodieCompactionOperation hc) Build Compaction operation payload from Avro version for using in Spark executors.
static HoodieCompactionOperation	buildFromFileSlice(String partitionPath, FileSlice fileSlice, Option<Function<Pair<String,FileSlice>,Map<String,Double>>> metricsCaptureFunction) Generate compaction operation from file-slice.
static HoodieCompactionPlan	buildFromFileSlices(List<Pair<String,FileSlice>> partitionFileSlicePairs, Option<Map<String,String>> extraMetadata, Option<Function<Pair<String,FileSlice>,Map<String,Double>>> metricsCaptureFunction) Generate compaction plan from file-slices.
static HoodieCompactionOperation	buildHoodieCompactionOperation(CompactionOperation op) Build Avro generated Compaction operation payload from compaction operation POJO for serialization.
static Map<HoodieFileGroupId,Pair<String,HoodieCompactionOperation>>	getAllPendingCompactionOperations(HoodieTableMetaClient metaClient) Get all PartitionPath + file-ids with pending Compaction operations and their target compaction instant time.
static List<Pair<HoodieInstant,HoodieCompactionPlan>>	getAllPendingCompactionPlans(HoodieTableMetaClient metaClient) Get all pending compaction plans along with their instants.
static HoodieCompactionPlan	getCompactionPlan(HoodieTableMetaClient metaClient, String compactionInstant)
static Option<Pair<HoodieTimeline,HoodieInstant>>	getDeltaCommitsSinceLatestCompaction(HoodieActiveTimeline activeTimeline) Returns a pair of (timeline containing the delta commits after the latest completed compaction commit, the completed compaction commit instant), if the latest completed compaction commit is present; a pair of (timeline containing all the delta commits, the first delta commit instant), if there is no completed compaction commit.
static Option<Pair<HoodieTimeline,HoodieInstant>>	getDeltaCommitsSinceLatestCompactionRequest(HoodieActiveTimeline activeTimeline)
static Option<HoodieInstant>	getOldestInstantToRetainForCompaction(HoodieActiveTimeline activeTimeline, int maxDeltaCommits) Gets the oldest instant to retain for MOR compaction.
static List<HoodieInstant>	getPendingCompactionInstantTimes(HoodieTableMetaClient metaClient) Return all pending compaction instant times.
static Stream<Pair<HoodieFileGroupId,Pair<String,HoodieCompactionOperation>>>	getPendingCompactionOperations(HoodieInstant instant, HoodieCompactionPlan compactionPlan)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

COMPACTION_METADATA_VERSION_1

public static final Integer COMPACTION_METADATA_VERSION_1

COMPACTION_METADATA_VERSION_2

public static final Integer COMPACTION_METADATA_VERSION_2

LATEST_COMPACTION_METADATA_VERSION

public static final Integer LATEST_COMPACTION_METADATA_VERSION

Constructor Detail

CompactionUtils

public CompactionUtils()

Method Detail

buildFromFileSlice

public static HoodieCompactionOperation buildFromFileSlice(String partitionPath,
                                                           FileSlice fileSlice,
                                                           Option<Function<Pair<String,FileSlice>,Map<String,Double>>> metricsCaptureFunction)

Generate compaction operation from file-slice.

Parameters:

partitionPath - Partition path

fileSlice - File Slice

metricsCaptureFunction - Metrics Capture function

Returns:

Compaction Operation

buildFromFileSlices

public static HoodieCompactionPlan buildFromFileSlices(List<Pair<String,FileSlice>> partitionFileSlicePairs,
                                                       Option<Map<String,String>> extraMetadata,
                                                       Option<Function<Pair<String,FileSlice>,Map<String,Double>>> metricsCaptureFunction)

Generate compaction plan from file-slices.

Parameters:

partitionFileSlicePairs - list of partition file-slice pairs

extraMetadata - Extra Metadata

metricsCaptureFunction - Metrics Capture function

buildHoodieCompactionOperation

public static HoodieCompactionOperation buildHoodieCompactionOperation(CompactionOperation op)

Build Avro generated Compaction operation payload from compaction operation POJO for serialization.

buildCompactionOperation

public static CompactionOperation buildCompactionOperation(HoodieCompactionOperation hc)

Build Compaction operation payload from Avro version for using in Spark executors.

Parameters:

hc - HoodieCompactionOperation

getAllPendingCompactionPlans

public static List<Pair<HoodieInstant,HoodieCompactionPlan>> getAllPendingCompactionPlans(HoodieTableMetaClient metaClient)

Get all pending compaction plans along with their instants.

Parameters:

metaClient - Hoodie Meta Client

getCompactionPlan

public static HoodieCompactionPlan getCompactionPlan(HoodieTableMetaClient metaClient,
                                                     String compactionInstant)
                                              throws IOException

Throws:

IOException

getAllPendingCompactionOperations

public static Map<HoodieFileGroupId,Pair<String,HoodieCompactionOperation>> getAllPendingCompactionOperations(HoodieTableMetaClient metaClient)

Get all PartitionPath + file-ids with pending Compaction operations and their target compaction instant time.

Parameters:

metaClient - Hoodie Table Meta Client

getPendingCompactionOperations

public static Stream<Pair<HoodieFileGroupId,Pair<String,HoodieCompactionOperation>>> getPendingCompactionOperations(HoodieInstant instant,
                                                                                                                    HoodieCompactionPlan compactionPlan)

getPendingCompactionInstantTimes

public static List<HoodieInstant> getPendingCompactionInstantTimes(HoodieTableMetaClient metaClient)

Return all pending compaction instant times.

Returns:

getDeltaCommitsSinceLatestCompaction

public static Option<Pair<HoodieTimeline,HoodieInstant>> getDeltaCommitsSinceLatestCompaction(HoodieActiveTimeline activeTimeline)

Returns a pair of (timeline containing the delta commits after the latest completed compaction commit, the completed compaction commit instant), if the latest completed compaction commit is present; a pair of (timeline containing all the delta commits, the first delta commit instant), if there is no completed compaction commit.

Parameters:

activeTimeline - Active timeline of a table.

Returns:

Pair of timeline containing delta commits and an instant.

getDeltaCommitsSinceLatestCompactionRequest

public static Option<Pair<HoodieTimeline,HoodieInstant>> getDeltaCommitsSinceLatestCompactionRequest(HoodieActiveTimeline activeTimeline)

getOldestInstantToRetainForCompaction

public static Option<HoodieInstant> getOldestInstantToRetainForCompaction(HoodieActiveTimeline activeTimeline,
                                                                          int maxDeltaCommits)

Gets the oldest instant to retain for MOR compaction. If there is no completed compaction, num delta commits >= "hoodie.compact.inline.max.delta.commits" If there is a completed compaction, num delta commits after latest completed compaction >= "hoodie.compact.inline.max.delta.commits"

Parameters:

activeTimeline - Active timeline of a table.

maxDeltaCommits - Maximum number of delta commits that trigger the compaction plan, i.e., "hoodie.compact.inline.max.delta.commits".

Returns:

the oldest instant to keep for MOR compaction.