org.apache.hudi.table.action.clean

Class CleanPlanner<T,I,K,O>

java.lang.Object

org.apache.hudi.table.action.clean.CleanPlanner<T,I,K,O>

All Implemented Interfaces:

Serializable

public class CleanPlanner<T,I,K,O>
extends Object
implements Serializable

Cleaner is responsible for garbage collecting older files in a given partition path. Such that

1) It provides sufficient time for existing queries running on older versions, to close

2) It bounds the growth of the files in the file system

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static Integer	CLEAN_PLAN_VERSION_1
static Integer	CLEAN_PLAN_VERSION_2
static Integer	LATEST_CLEAN_PLAN_VERSION

Constructor Summary

Constructors

Constructor and Description
CleanPlanner(HoodieEngineContext context, HoodieTable<T,I,K,O> hoodieTable, HoodieWriteConfig config)

Method Summary

Modifier and Type	Method and Description
Pair<Boolean,List<CleanFileInfo>>	getDeletePaths(String partitionPath, Option<HoodieInstant> earliestCommitToRetain) Returns files to be cleaned for the given partitionPath based on cleaning policy.
Option<HoodieInstant>	getEarliestCommitToRetain() Returns earliest commit to retain based on cleaning policy.
String	getLastCompletedCommitTimestamp() Returns the last completed commit timestamp before clean.
List<String>	getPartitionPathsToClean(Option<HoodieInstant> earliestRetainedInstant) Returns list of partitions where clean operations needs to be performed.
Stream<String>	getSavepointedDataFiles(String savepointTime) Get the list of data file names savepointed.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CLEAN_PLAN_VERSION_1

public static final Integer CLEAN_PLAN_VERSION_1

CLEAN_PLAN_VERSION_2

public static final Integer CLEAN_PLAN_VERSION_2

LATEST_CLEAN_PLAN_VERSION

public static final Integer LATEST_CLEAN_PLAN_VERSION

Constructor Detail

CleanPlanner

public CleanPlanner(HoodieEngineContext context,
                    HoodieTable<T,I,K,O> hoodieTable,
                    HoodieWriteConfig config)

Method Detail

getSavepointedDataFiles

public Stream<String> getSavepointedDataFiles(String savepointTime)

Get the list of data file names savepointed.

getPartitionPathsToClean

public List<String> getPartitionPathsToClean(Option<HoodieInstant> earliestRetainedInstant)
                                      throws IOException

Returns list of partitions where clean operations needs to be performed.

Parameters:

earliestRetainedInstant - New instant to be retained after this cleanup operation

Returns:

list of partitions to scan for cleaning

Throws:

IOException - when underlying file-system throws this exception

getDeletePaths

public Pair<Boolean,List<CleanFileInfo>> getDeletePaths(String partitionPath,
                                                        Option<HoodieInstant> earliestCommitToRetain)

Returns files to be cleaned for the given partitionPath based on cleaning policy.

getEarliestCommitToRetain

public Option<HoodieInstant> getEarliestCommitToRetain()

Returns earliest commit to retain based on cleaning policy.

getLastCompletedCommitTimestamp

public String getLastCompletedCommitTimestamp()

Returns the last completed commit timestamp before clean.