org.apache.hudi.config

Class HoodieClusteringConfig

java.lang.Object

org.apache.hudi.common.config.HoodieConfig

org.apache.hudi.config.HoodieClusteringConfig

All Implemented Interfaces:

Serializable

public class HoodieClusteringConfig
extends HoodieConfig

Clustering specific configs.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HoodieClusteringConfig.Builder
static class	HoodieClusteringConfig.LayoutOptimizationStrategy Layout optimization strategies such as Z-order/Hilbert space-curves, etc
static class	HoodieClusteringConfig.SpatialCurveCompositionStrategyType Type of a strategy for building Z-order/Hilbert space-filling curves.

Field Summary

Fields

Modifier and Type	Field and Description
static ConfigProperty<String>	ASYNC_CLUSTERING_ENABLE
static String	ASYNC_CLUSTERING_ENABLE_OPT_KEY Deprecated. Use ASYNC_CLUSTERING_ENABLE and its methods instead
static ConfigProperty<String>	ASYNC_CLUSTERING_MAX_COMMITS
static String	CLUSTERING_EXECUTION_STRATEGY_CLASS Deprecated. Use EXECUTION_STRATEGY_CLASS_NAME and its methods instead
static String	CLUSTERING_MAX_BYTES_PER_GROUP Deprecated. Use PLAN_STRATEGY_MAX_BYTES_PER_OUTPUT_FILEGROUP and its methods instead
static String	CLUSTERING_MAX_NUM_GROUPS Deprecated. Use PLAN_STRATEGY_MAX_GROUPS and its methods instead
static String	CLUSTERING_PLAN_SMALL_FILE_LIMIT Deprecated. Use PLAN_STRATEGY_SMALL_FILE_LIMIT and its methods instead
static String	CLUSTERING_PLAN_STRATEGY_CLASS Deprecated. Use PLAN_STRATEGY_CLASS_NAME and its methods instead
static String	CLUSTERING_SORT_COLUMNS_PROPERTY Deprecated. Use PLAN_STRATEGY_SORT_COLUMNS and its methods instead
static String	CLUSTERING_STRATEGY_PARAM_PREFIX
static String	CLUSTERING_TARGET_FILE_MAX_BYTES Deprecated. Use PLAN_STRATEGY_TARGET_FILE_MAX_BYTES and its methods instead
static String	CLUSTERING_TARGET_PARTITIONS Deprecated. Use DAYBASED_LOOKBACK_PARTITIONS and its methods instead
static String	CLUSTERING_UPDATES_STRATEGY_PROP Deprecated. Use UPDATES_STRATEGY and its methods instead
static ConfigProperty<String>	DAYBASED_LOOKBACK_PARTITIONS
static String	DEFAULT_ASYNC_CLUSTERING_ENABLE_OPT_VAL Deprecated. Use ASYNC_CLUSTERING_ENABLE and its methods instead
static String	DEFAULT_CLUSTERING_EXECUTION_STRATEGY_CLASS Deprecated. Use EXECUTION_STRATEGY_CLASS_NAME and its methods instead
static String	DEFAULT_CLUSTERING_MAX_GROUP_SIZE Deprecated. Use PLAN_STRATEGY_MAX_BYTES_PER_OUTPUT_FILEGROUP and its methods instead
static String	DEFAULT_CLUSTERING_MAX_NUM_GROUPS Deprecated. Use PLAN_STRATEGY_MAX_GROUPS and its methods instead
static String	DEFAULT_CLUSTERING_PLAN_SMALL_FILE_LIMIT Deprecated. Use PLAN_STRATEGY_SMALL_FILE_LIMIT and its methods instead
static String	DEFAULT_CLUSTERING_PLAN_STRATEGY_CLASS Deprecated. Use PLAN_STRATEGY_CLASS_NAME and its methods instead
static String	DEFAULT_CLUSTERING_TARGET_FILE_MAX_BYTES Deprecated. Use PLAN_STRATEGY_TARGET_FILE_MAX_BYTES and its methods instead
static String	DEFAULT_CLUSTERING_TARGET_PARTITIONS Deprecated. Use DAYBASED_LOOKBACK_PARTITIONS and its methods instead
static String	DEFAULT_CLUSTERING_UPDATES_STRATEGY Deprecated. Use UPDATES_STRATEGY and its methods instead
static ConfigProperty<String>	EXECUTION_STRATEGY_CLASS_NAME
static String	FLINK_SIZED_BASED_CLUSTERING_PLAN_STRATEGY
static ConfigProperty<String>	INLINE_CLUSTERING
static String	INLINE_CLUSTERING_MAX_COMMIT_PROP Deprecated. Use INLINE_CLUSTERING_MAX_COMMITS and its methods instead
static ConfigProperty<String>	INLINE_CLUSTERING_MAX_COMMITS
static String	INLINE_CLUSTERING_PROP Deprecated. Use INLINE_CLUSTERING and its methods instead
static String	JAVA_SIZED_BASED_CLUSTERING_PLAN_STRATEGY
static String	JAVA_SORT_AND_SIZE_EXECUTION_STRATEGY
static ConfigProperty<String>	LAYOUT_OPTIMIZE_BUILD_CURVE_SAMPLE_SIZE NOTE: This setting only has effect if LAYOUT_OPTIMIZE_SPATIAL_CURVE_BUILD_METHOD value is set to "sample" Determines target sample size used by the Boundary-based Interleaved Index method.
static ConfigProperty	LAYOUT_OPTIMIZE_DATA_SKIPPING_ENABLE Deprecated. this setting has no effect
static ConfigProperty	LAYOUT_OPTIMIZE_ENABLE Deprecated. this setting has no effect. Please refer to clustering configuration, as well as LAYOUT_OPTIMIZE_STRATEGY config to enable advanced record layout optimization strategies
static ConfigProperty<String>	LAYOUT_OPTIMIZE_SPATIAL_CURVE_BUILD_METHOD NOTE: This setting only has effect if LAYOUT_OPTIMIZE_STRATEGY value is set to either "z-order" or "hilbert" (ie leveraging space-filling curves) Currently, two methods to order records along the curve are supported "build" and "sample": Direct: entails that spatial curve will be built in full, "filling in" all of the individual points corresponding to each individual record Sample: leverages boundary-base interleaved index method (described in more details in Amazon DynamoDB blog [1]) NOTE: Boundary-based interleaved Index method has better generalization, but is slower than direct method.
static ConfigProperty<String>	LAYOUT_OPTIMIZE_STRATEGY Determines ordering strategy in for records layout optimization.
static ConfigProperty<String>	PARTITION_FILTER_BEGIN_PARTITION
static ConfigProperty<String>	PARTITION_FILTER_END_PARTITION
static ConfigProperty<String>	PARTITION_REGEX_PATTERN
static ConfigProperty<String>	PARTITION_SELECTED
static String	PLAN_PARTITION_FILTER_MODE
static ConfigProperty<ClusteringPlanPartitionFilterMode>	PLAN_PARTITION_FILTER_MODE_NAME
static ConfigProperty<String>	PLAN_STRATEGY_CLASS_NAME
static ConfigProperty<String>	PLAN_STRATEGY_MAX_BYTES_PER_OUTPUT_FILEGROUP
static ConfigProperty<String>	PLAN_STRATEGY_MAX_GROUPS
static ConfigProperty<String>	PLAN_STRATEGY_SKIP_PARTITIONS_FROM_LATEST
static ConfigProperty<String>	PLAN_STRATEGY_SMALL_FILE_LIMIT
static ConfigProperty<String>	PLAN_STRATEGY_SORT_COLUMNS
static ConfigProperty<String>	PLAN_STRATEGY_TARGET_FILE_MAX_BYTES
static ConfigProperty<Boolean>	PRESERVE_COMMIT_METADATA
static ConfigProperty<Boolean>	ROLLBACK_PENDING_CLUSTERING_ON_CONFLICT
static ConfigProperty<String>	SCHEDULE_INLINE_CLUSTERING
static String	SPARK_SIZED_BASED_CLUSTERING_PLAN_STRATEGY
static String	SPARK_SORT_AND_SIZE_EXECUTION_STRATEGY
static ConfigProperty<String>	UPDATES_STRATEGY

Fields inherited from class org.apache.hudi.common.config.HoodieConfig

CONFIG_VALUES_DELIMITER, props

Constructor Summary

Constructors

Constructor and Description
HoodieClusteringConfig()

Method Summary

Modifier and Type	Method and Description
static HoodieClusteringConfig	from(TypedProperties props)
boolean	isAsyncClusteringEnabled()
boolean	isInlineClusteringEnabled()
static HoodieClusteringConfig.Builder	newBuilder()

Methods inherited from class org.apache.hudi.common.config.HoodieConfig

contains, contains, create, getBoolean, getBooleanOrDefault, getBooleanOrDefault, getDouble, getFloat, getInt, getIntOrDefault, getLong, getProps, getProps, getSplitStrings, getSplitStrings, getString, getString, getStringOrDefault, getStringOrDefault, getStringOrThrow, setAll, setDefault, setDefaultOnCondition, setDefaults, setDefaultValue, setDefaultValue, setValue, setValue

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CLUSTERING_STRATEGY_PARAM_PREFIX

public static final String CLUSTERING_STRATEGY_PARAM_PREFIX

See Also:

Constant Field Values

SPARK_SIZED_BASED_CLUSTERING_PLAN_STRATEGY

public static final String SPARK_SIZED_BASED_CLUSTERING_PLAN_STRATEGY

See Also:

Constant Field Values

FLINK_SIZED_BASED_CLUSTERING_PLAN_STRATEGY

public static final String FLINK_SIZED_BASED_CLUSTERING_PLAN_STRATEGY

See Also:

Constant Field Values

JAVA_SIZED_BASED_CLUSTERING_PLAN_STRATEGY

public static final String JAVA_SIZED_BASED_CLUSTERING_PLAN_STRATEGY

See Also:

Constant Field Values

SPARK_SORT_AND_SIZE_EXECUTION_STRATEGY

public static final String SPARK_SORT_AND_SIZE_EXECUTION_STRATEGY

See Also:

Constant Field Values

JAVA_SORT_AND_SIZE_EXECUTION_STRATEGY

public static final String JAVA_SORT_AND_SIZE_EXECUTION_STRATEGY

See Also:

Constant Field Values

PLAN_PARTITION_FILTER_MODE

public static final String PLAN_PARTITION_FILTER_MODE

See Also:

Constant Field Values

DAYBASED_LOOKBACK_PARTITIONS

public static final ConfigProperty<String> DAYBASED_LOOKBACK_PARTITIONS

PARTITION_FILTER_BEGIN_PARTITION

public static final ConfigProperty<String> PARTITION_FILTER_BEGIN_PARTITION

PARTITION_FILTER_END_PARTITION

public static final ConfigProperty<String> PARTITION_FILTER_END_PARTITION

PLAN_STRATEGY_SMALL_FILE_LIMIT

public static final ConfigProperty<String> PLAN_STRATEGY_SMALL_FILE_LIMIT

PARTITION_REGEX_PATTERN

public static final ConfigProperty<String> PARTITION_REGEX_PATTERN

PARTITION_SELECTED

public static final ConfigProperty<String> PARTITION_SELECTED

PLAN_STRATEGY_CLASS_NAME

public static final ConfigProperty<String> PLAN_STRATEGY_CLASS_NAME

EXECUTION_STRATEGY_CLASS_NAME

public static final ConfigProperty<String> EXECUTION_STRATEGY_CLASS_NAME

INLINE_CLUSTERING

public static final ConfigProperty<String> INLINE_CLUSTERING

INLINE_CLUSTERING_MAX_COMMITS

public static final ConfigProperty<String> INLINE_CLUSTERING_MAX_COMMITS

ASYNC_CLUSTERING_MAX_COMMITS

public static final ConfigProperty<String> ASYNC_CLUSTERING_MAX_COMMITS

PLAN_STRATEGY_SKIP_PARTITIONS_FROM_LATEST

public static final ConfigProperty<String> PLAN_STRATEGY_SKIP_PARTITIONS_FROM_LATEST

PLAN_PARTITION_FILTER_MODE_NAME

public static final ConfigProperty<ClusteringPlanPartitionFilterMode> PLAN_PARTITION_FILTER_MODE_NAME

PLAN_STRATEGY_MAX_BYTES_PER_OUTPUT_FILEGROUP

public static final ConfigProperty<String> PLAN_STRATEGY_MAX_BYTES_PER_OUTPUT_FILEGROUP

PLAN_STRATEGY_MAX_GROUPS

public static final ConfigProperty<String> PLAN_STRATEGY_MAX_GROUPS

PLAN_STRATEGY_TARGET_FILE_MAX_BYTES

public static final ConfigProperty<String> PLAN_STRATEGY_TARGET_FILE_MAX_BYTES

PLAN_STRATEGY_SORT_COLUMNS

public static final ConfigProperty<String> PLAN_STRATEGY_SORT_COLUMNS

UPDATES_STRATEGY

public static final ConfigProperty<String> UPDATES_STRATEGY

SCHEDULE_INLINE_CLUSTERING

public static final ConfigProperty<String> SCHEDULE_INLINE_CLUSTERING

ASYNC_CLUSTERING_ENABLE

public static final ConfigProperty<String> ASYNC_CLUSTERING_ENABLE

PRESERVE_COMMIT_METADATA

public static final ConfigProperty<Boolean> PRESERVE_COMMIT_METADATA

LAYOUT_OPTIMIZE_ENABLE

public static final ConfigProperty LAYOUT_OPTIMIZE_ENABLE

Deprecated. this setting has no effect. Please refer to clustering configuration, as well as LAYOUT_OPTIMIZE_STRATEGY config to enable advanced record layout optimization strategies

LAYOUT_OPTIMIZE_STRATEGY

public static final ConfigProperty<String> LAYOUT_OPTIMIZE_STRATEGY

Determines ordering strategy in for records layout optimization. Currently, following strategies are supported

• Linear: simply orders records lexicographically
• Z-order: orders records along Z-order spatial-curve
• Hilbert: orders records along Hilbert's spatial-curve

NOTE: "z-order", "hilbert" strategies may consume considerably more compute, than "linear". Make sure to perform small-scale local testing for your dataset before applying globally.

LAYOUT_OPTIMIZE_SPATIAL_CURVE_BUILD_METHOD

public static final ConfigProperty<String> LAYOUT_OPTIMIZE_SPATIAL_CURVE_BUILD_METHOD

NOTE: This setting only has effect if LAYOUT_OPTIMIZE_STRATEGY value is set to either "z-order" or "hilbert" (ie leveraging space-filling curves) Currently, two methods to order records along the curve are supported "build" and "sample":

• Direct: entails that spatial curve will be built in full, "filling in" all of the individual points corresponding to each individual record
• Sample: leverages boundary-base interleaved index method (described in more details in Amazon DynamoDB blog [1])

NOTE: Boundary-based interleaved Index method has better generalization, but is slower than direct method. Please refer to RFC-28 for specific elaboration on both flows. [1] https://aws.amazon.com/cn/blogs/database/tag/z-order/

LAYOUT_OPTIMIZE_BUILD_CURVE_SAMPLE_SIZE

public static final ConfigProperty<String> LAYOUT_OPTIMIZE_BUILD_CURVE_SAMPLE_SIZE

NOTE: This setting only has effect if LAYOUT_OPTIMIZE_SPATIAL_CURVE_BUILD_METHOD value is set to "sample" Determines target sample size used by the Boundary-based Interleaved Index method. Larger sample size entails better layout optimization outcomes, at the expense of higher memory footprint.

LAYOUT_OPTIMIZE_DATA_SKIPPING_ENABLE

public static final ConfigProperty LAYOUT_OPTIMIZE_DATA_SKIPPING_ENABLE

Deprecated. this setting has no effect

ROLLBACK_PENDING_CLUSTERING_ON_CONFLICT

public static final ConfigProperty<Boolean> ROLLBACK_PENDING_CLUSTERING_ON_CONFLICT

CLUSTERING_PLAN_STRATEGY_CLASS

@Deprecated
public static final String CLUSTERING_PLAN_STRATEGY_CLASS

Deprecated. Use PLAN_STRATEGY_CLASS_NAME and its methods instead

DEFAULT_CLUSTERING_PLAN_STRATEGY_CLASS

@Deprecated
public static final String DEFAULT_CLUSTERING_PLAN_STRATEGY_CLASS

Deprecated. Use PLAN_STRATEGY_CLASS_NAME and its methods instead

CLUSTERING_EXECUTION_STRATEGY_CLASS

@Deprecated
public static final String CLUSTERING_EXECUTION_STRATEGY_CLASS

Deprecated. Use EXECUTION_STRATEGY_CLASS_NAME and its methods instead

DEFAULT_CLUSTERING_EXECUTION_STRATEGY_CLASS

@Deprecated
public static final String DEFAULT_CLUSTERING_EXECUTION_STRATEGY_CLASS

Deprecated. Use EXECUTION_STRATEGY_CLASS_NAME and its methods instead

INLINE_CLUSTERING_PROP

@Deprecated
public static final String INLINE_CLUSTERING_PROP

Deprecated. Use INLINE_CLUSTERING and its methods instead

INLINE_CLUSTERING_MAX_COMMIT_PROP

@Deprecated
public static final String INLINE_CLUSTERING_MAX_COMMIT_PROP

Deprecated. Use INLINE_CLUSTERING_MAX_COMMITS and its methods instead

CLUSTERING_TARGET_PARTITIONS

@Deprecated
public static final String CLUSTERING_TARGET_PARTITIONS

Deprecated. Use DAYBASED_LOOKBACK_PARTITIONS and its methods instead

DEFAULT_CLUSTERING_TARGET_PARTITIONS

@Deprecated
public static final String DEFAULT_CLUSTERING_TARGET_PARTITIONS

Deprecated. Use DAYBASED_LOOKBACK_PARTITIONS and its methods instead

CLUSTERING_PLAN_SMALL_FILE_LIMIT

@Deprecated
public static final String CLUSTERING_PLAN_SMALL_FILE_LIMIT

Deprecated. Use PLAN_STRATEGY_SMALL_FILE_LIMIT and its methods instead

DEFAULT_CLUSTERING_PLAN_SMALL_FILE_LIMIT

@Deprecated
public static final String DEFAULT_CLUSTERING_PLAN_SMALL_FILE_LIMIT

Deprecated. Use PLAN_STRATEGY_SMALL_FILE_LIMIT and its methods instead

CLUSTERING_MAX_BYTES_PER_GROUP

@Deprecated
public static final String CLUSTERING_MAX_BYTES_PER_GROUP

Deprecated. Use PLAN_STRATEGY_MAX_BYTES_PER_OUTPUT_FILEGROUP and its methods instead

DEFAULT_CLUSTERING_MAX_GROUP_SIZE

@Deprecated
public static final String DEFAULT_CLUSTERING_MAX_GROUP_SIZE

Deprecated. Use PLAN_STRATEGY_MAX_BYTES_PER_OUTPUT_FILEGROUP and its methods instead

CLUSTERING_MAX_NUM_GROUPS

@Deprecated
public static final String CLUSTERING_MAX_NUM_GROUPS

Deprecated. Use PLAN_STRATEGY_MAX_GROUPS and its methods instead

DEFAULT_CLUSTERING_MAX_NUM_GROUPS

@Deprecated
public static final String DEFAULT_CLUSTERING_MAX_NUM_GROUPS

Deprecated. Use PLAN_STRATEGY_MAX_GROUPS and its methods instead

CLUSTERING_TARGET_FILE_MAX_BYTES

@Deprecated
public static final String CLUSTERING_TARGET_FILE_MAX_BYTES

Deprecated. Use PLAN_STRATEGY_TARGET_FILE_MAX_BYTES and its methods instead

DEFAULT_CLUSTERING_TARGET_FILE_MAX_BYTES

@Deprecated
public static final String DEFAULT_CLUSTERING_TARGET_FILE_MAX_BYTES

Deprecated. Use PLAN_STRATEGY_TARGET_FILE_MAX_BYTES and its methods instead

CLUSTERING_SORT_COLUMNS_PROPERTY

@Deprecated
public static final String CLUSTERING_SORT_COLUMNS_PROPERTY

Deprecated. Use PLAN_STRATEGY_SORT_COLUMNS and its methods instead

CLUSTERING_UPDATES_STRATEGY_PROP

@Deprecated
public static final String CLUSTERING_UPDATES_STRATEGY_PROP

Deprecated. Use UPDATES_STRATEGY and its methods instead

DEFAULT_CLUSTERING_UPDATES_STRATEGY

@Deprecated
public static final String DEFAULT_CLUSTERING_UPDATES_STRATEGY

Deprecated. Use UPDATES_STRATEGY and its methods instead

ASYNC_CLUSTERING_ENABLE_OPT_KEY

@Deprecated
public static final String ASYNC_CLUSTERING_ENABLE_OPT_KEY

Deprecated. Use ASYNC_CLUSTERING_ENABLE and its methods instead

DEFAULT_ASYNC_CLUSTERING_ENABLE_OPT_VAL

@Deprecated
public static final String DEFAULT_ASYNC_CLUSTERING_ENABLE_OPT_VAL

Deprecated. Use ASYNC_CLUSTERING_ENABLE and its methods instead

Constructor Detail

HoodieClusteringConfig

public HoodieClusteringConfig()

Method Detail

isAsyncClusteringEnabled

public boolean isAsyncClusteringEnabled()

isInlineClusteringEnabled

public boolean isInlineClusteringEnabled()

from

public static HoodieClusteringConfig from(TypedProperties props)

newBuilder

public static HoodieClusteringConfig.Builder newBuilder()