org.apache.hudi.table.action.compact.strategy

Class DayBasedCompactionStrategy

java.lang.Object

org.apache.hudi.table.action.compact.strategy.CompactionStrategy

org.apache.hudi.table.action.compact.strategy.DayBasedCompactionStrategy

All Implemented Interfaces:

Serializable

Direct Known Subclasses:

BoundedPartitionAwareCompactionStrategy

public class DayBasedCompactionStrategy
extends CompactionStrategy

This strategy orders compactions in reverse order of creation of Hive Partitions. It helps to compact data in latest partitions first and then older capped at the Total_IO allowed.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected static Comparator<String>	comparator
protected static String	DATE_PARTITION_FORMAT

Fields inherited from class org.apache.hudi.table.action.compact.strategy.CompactionStrategy

TOTAL_IO_MB, TOTAL_IO_READ_MB, TOTAL_IO_WRITE_MB, TOTAL_LOG_FILE_SIZE, TOTAL_LOG_FILES

Constructor Summary

Constructors

Constructor and Description
DayBasedCompactionStrategy()

Method Summary

Modifier and Type	Method and Description
List<String>	filterPartitionPaths(HoodieWriteConfig writeConfig, List<String> allPartitionPaths) Filter the partition paths based on compaction strategy.
Comparator<String>	getComparator()
protected static String	getPartitionPathWithoutPartitionKeys(String partitionPath) If is Hive style partition path, convert it to regular partition path.
List<HoodieCompactionOperation>	orderAndFilter(HoodieWriteConfig writeConfig, List<HoodieCompactionOperation> operations, List<HoodieCompactionPlan> pendingCompactionPlans) Order and Filter the list of compactions.

Methods inherited from class org.apache.hudi.table.action.compact.strategy.CompactionStrategy

captureMetrics, generateCompactionPlan

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DATE_PARTITION_FORMAT

protected static final String DATE_PARTITION_FORMAT

See Also:

Constant Field Values

comparator

protected static Comparator<String> comparator

Constructor Detail

DayBasedCompactionStrategy

public DayBasedCompactionStrategy()

Method Detail

getComparator

public Comparator<String> getComparator()

orderAndFilter

public List<HoodieCompactionOperation> orderAndFilter(HoodieWriteConfig writeConfig,
                                                      List<HoodieCompactionOperation> operations,
                                                      List<HoodieCompactionPlan> pendingCompactionPlans)

Description copied from class: CompactionStrategy

Order and Filter the list of compactions. Use the metrics captured with the captureMetrics to order and filter out compactions

Overrides:

orderAndFilter in class CompactionStrategy

Parameters:

writeConfig - config for this compaction is passed in

operations - list of compactions collected

pendingCompactionPlans - Pending Compaction Plans for strategy to schedule next compaction plan

Returns:

list of compactions to perform in this run

filterPartitionPaths

public List<String> filterPartitionPaths(HoodieWriteConfig writeConfig,
                                         List<String> allPartitionPaths)

Description copied from class: CompactionStrategy

Filter the partition paths based on compaction strategy.

Overrides:

filterPartitionPaths in class CompactionStrategy

Returns:

getPartitionPathWithoutPartitionKeys

protected static String getPartitionPathWithoutPartitionKeys(String partitionPath)

If is Hive style partition path, convert it to regular partition path. e.g. year=2019/month=11/day=24 => 2019/11/24