org.apache.hudi.table.action.compact.strategy

Class BoundedPartitionAwareCompactionStrategy

java.lang.Object

org.apache.hudi.table.action.compact.strategy.CompactionStrategy

org.apache.hudi.table.action.compact.strategy.DayBasedCompactionStrategy

org.apache.hudi.table.action.compact.strategy.BoundedPartitionAwareCompactionStrategy

All Implemented Interfaces:

Serializable

public class BoundedPartitionAwareCompactionStrategy
extends DayBasedCompactionStrategy

This strategy ensures that the last N partitions are picked up even if there are later partitions created for the table. lastNPartitions is defined as the N partitions before the currentDate. currentDay = 2018/01/01 The table has partitions for 2018/02/02 and 2018/03/03 beyond the currentDay This strategy will pick up the following partitions for compaction : (2018/01/01, allPartitionsInRange[(2018/01/01 - lastNPartitions) to 2018/01/01), 2018/02/02, 2018/03/03)

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.hudi.table.action.compact.strategy.DayBasedCompactionStrategy

comparator, DATE_PARTITION_FORMAT

Fields inherited from class org.apache.hudi.table.action.compact.strategy.CompactionStrategy

TOTAL_IO_MB, TOTAL_IO_READ_MB, TOTAL_IO_WRITE_MB, TOTAL_LOG_FILE_SIZE, TOTAL_LOG_FILES

Constructor Summary

Constructors

Constructor and Description
BoundedPartitionAwareCompactionStrategy()

Method Summary

Modifier and Type	Method and Description
List<String>	filterPartitionPaths(HoodieWriteConfig writeConfig, List<String> partitionPaths) Filter the partition paths based on compaction strategy.
static Date	getDateAtOffsetFromToday(int offset)
List<HoodieCompactionOperation>	orderAndFilter(HoodieWriteConfig writeConfig, List<HoodieCompactionOperation> operations, List<HoodieCompactionPlan> pendingCompactionPlans) Order and Filter the list of compactions.

Methods inherited from class org.apache.hudi.table.action.compact.strategy.DayBasedCompactionStrategy

getComparator, getPartitionPathWithoutPartitionKeys

Methods inherited from class org.apache.hudi.table.action.compact.strategy.CompactionStrategy

captureMetrics, generateCompactionPlan

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BoundedPartitionAwareCompactionStrategy

public BoundedPartitionAwareCompactionStrategy()

Method Detail

orderAndFilter

public List<HoodieCompactionOperation> orderAndFilter(HoodieWriteConfig writeConfig,
                                                      List<HoodieCompactionOperation> operations,
                                                      List<HoodieCompactionPlan> pendingCompactionPlans)

Description copied from class: CompactionStrategy

Order and Filter the list of compactions. Use the metrics captured with the captureMetrics to order and filter out compactions

Overrides:

orderAndFilter in class DayBasedCompactionStrategy

Parameters:

writeConfig - config for this compaction is passed in

operations - list of compactions collected

pendingCompactionPlans - Pending Compaction Plans for strategy to schedule next compaction plan

Returns:

list of compactions to perform in this run

filterPartitionPaths

public List<String> filterPartitionPaths(HoodieWriteConfig writeConfig,
                                         List<String> partitionPaths)

Description copied from class: CompactionStrategy

Filter the partition paths based on compaction strategy.

Overrides:

filterPartitionPaths in class DayBasedCompactionStrategy

Returns:

getDateAtOffsetFromToday

public static Date getDateAtOffsetFromToday(int offset)