org.apache.hudi.table.action.cluster.strategy

Class PartitionAwareClusteringPlanStrategy<T,I,K,O>

java.lang.Object

org.apache.hudi.table.action.cluster.strategy.ClusteringPlanStrategy<T,I,K,O>

org.apache.hudi.table.action.cluster.strategy.PartitionAwareClusteringPlanStrategy<T,I,K,O>

All Implemented Interfaces:

Serializable

Direct Known Subclasses:

BaseConsistentHashingBucketClusteringPlanStrategy

public abstract class PartitionAwareClusteringPlanStrategy<T,I,K,O>
extends ClusteringPlanStrategy<T,I,K,O>

Scheduling strategy with restriction that clustering groups can only contain files from same partition.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.hudi.table.action.cluster.strategy.ClusteringPlanStrategy

CLUSTERING_PLAN_VERSION_1

Constructor Summary

Constructors

Constructor and Description
PartitionAwareClusteringPlanStrategy(HoodieTable table, HoodieEngineContext engineContext, HoodieWriteConfig writeConfig)

Method Summary

Modifier and Type	Method and Description
protected Stream<HoodieClusteringGroup>	buildClusteringGroupsForPartition(String partitionPath, List<FileSlice> fileSlices) Create Clustering group based on files eligible for clustering in the partition.
protected List<String>	filterPartitionPaths(List<String> partitionPaths) Return list of partition paths to be considered for clustering.
Option<HoodieClusteringPlan>	generateClusteringPlan() Generate metadata for grouping eligible files and create a plan.
protected int	getNumberOfOutputFileGroups(long groupSize, long targetFileSize)
List<String>	getRegexPatternMatchedPartitions(HoodieWriteConfig config, List<String> partitionPaths)

Methods inherited from class org.apache.hudi.table.action.cluster.strategy.ClusteringPlanStrategy

buildMetrics, checkAndGetClusteringPlanStrategy, checkPrecondition, getEngineContext, getExtraMetadata, getFileSliceInfo, getFileSlicesEligibleForClustering, getHoodieTable, getPlanVersion, getStrategyParams, getWriteConfig

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PartitionAwareClusteringPlanStrategy

public PartitionAwareClusteringPlanStrategy(HoodieTable table,
                                            HoodieEngineContext engineContext,
                                            HoodieWriteConfig writeConfig)

Method Detail

buildClusteringGroupsForPartition

protected Stream<HoodieClusteringGroup> buildClusteringGroupsForPartition(String partitionPath,
                                                                          List<FileSlice> fileSlices)

Create Clustering group based on files eligible for clustering in the partition.

filterPartitionPaths

protected List<String> filterPartitionPaths(List<String> partitionPaths)

Return list of partition paths to be considered for clustering.

generateClusteringPlan

public Option<HoodieClusteringPlan> generateClusteringPlan()

Description copied from class: ClusteringPlanStrategy

Generate metadata for grouping eligible files and create a plan. Note that data is not moved around as part of this step. If there is no data available to cluster, return None.

Specified by:

generateClusteringPlan in class ClusteringPlanStrategy<T,I,K,O>

getRegexPatternMatchedPartitions

public List<String> getRegexPatternMatchedPartitions(HoodieWriteConfig config,
                                                     List<String> partitionPaths)

getNumberOfOutputFileGroups

protected int getNumberOfOutputFileGroups(long groupSize,
                                          long targetFileSize)