org.apache.hudi.table.action.cluster.strategy

Class ClusteringPlanStrategy<T extends HoodieRecordPayload,I,K,O>

java.lang.Object

org.apache.hudi.table.action.cluster.strategy.ClusteringPlanStrategy<T,I,K,O>

All Implemented Interfaces:

Serializable

Direct Known Subclasses:

PartitionAwareClusteringPlanStrategy

public abstract class ClusteringPlanStrategy<T extends HoodieRecordPayload,I,K,O>
extends Object
implements Serializable

Pluggable implementation for scheduling clustering and creating ClusteringPlan.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static int	CLUSTERING_PLAN_VERSION_1

Constructor Summary

Constructors

Constructor and Description
ClusteringPlanStrategy(HoodieTable table, HoodieEngineContext engineContext, HoodieWriteConfig writeConfig)

Method Summary

Modifier and Type	Method and Description
protected Map<String,Double>	buildMetrics(List<FileSlice> fileSlices) Generate metrics for the data to be clustered.
static String	checkAndGetClusteringPlanStrategy(HoodieWriteConfig config) Check if the given class is deprecated.
abstract Option<HoodieClusteringPlan>	generateClusteringPlan() Generate metadata for grouping eligible files and create a plan.
protected HoodieEngineContext	getEngineContext()
protected Map<String,String>	getExtraMetadata() Returns any specific parameters to be stored as part of clustering metadata.
protected static List<HoodieSliceInfo>	getFileSliceInfo(List<FileSlice> slices) Transform FileSlice to HoodieSliceInfo.
protected Stream<FileSlice>	getFileSlicesEligibleForClustering(String partition) Return file slices eligible for clustering.
protected HoodieTable<T,I,K,O>	getHoodieTable()
protected int	getPlanVersion() Version to support future changes for plan.
protected abstract Map<String,String>	getStrategyParams() Get parameters specific to strategy.
protected HoodieWriteConfig	getWriteConfig()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CLUSTERING_PLAN_VERSION_1

public static final int CLUSTERING_PLAN_VERSION_1

See Also:

Constant Field Values

Constructor Detail

ClusteringPlanStrategy

public ClusteringPlanStrategy(HoodieTable table,
                              HoodieEngineContext engineContext,
                              HoodieWriteConfig writeConfig)

Method Detail

checkAndGetClusteringPlanStrategy

public static String checkAndGetClusteringPlanStrategy(HoodieWriteConfig config)

Check if the given class is deprecated. If it is, then try to convert it to suitable one and update the write config accordingly.

Parameters:

config - write config

Returns:

class name of clustering plan strategy

generateClusteringPlan

public abstract Option<HoodieClusteringPlan> generateClusteringPlan()

Generate metadata for grouping eligible files and create a plan. Note that data is not moved around as part of this step. If there is no data available to cluster, return None.

getFileSlicesEligibleForClustering

protected Stream<FileSlice> getFileSlicesEligibleForClustering(String partition)

Return file slices eligible for clustering. FileIds in pending clustering/compaction are not eligible for clustering.

getStrategyParams

protected abstract Map<String,String> getStrategyParams()

Get parameters specific to strategy. These parameters are passed from 'schedule clustering' step to 'execute clustering' step. 'execute clustering' step is typically async. So these params help with passing any required context from schedule to run step.

getExtraMetadata

protected Map<String,String> getExtraMetadata()

Returns any specific parameters to be stored as part of clustering metadata.

getPlanVersion

protected int getPlanVersion()

Version to support future changes for plan.

getFileSliceInfo

protected static List<HoodieSliceInfo> getFileSliceInfo(List<FileSlice> slices)

Transform FileSlice to HoodieSliceInfo.

buildMetrics

protected Map<String,Double> buildMetrics(List<FileSlice> fileSlices)

Generate metrics for the data to be clustered.

getHoodieTable

protected HoodieTable<T,I,K,O> getHoodieTable()

getEngineContext

protected HoodieEngineContext getEngineContext()

getWriteConfig

protected HoodieWriteConfig getWriteConfig()