org.apache.hudi.common.table.timeline

Class HoodieDefaultTimeline

java.lang.Object

org.apache.hudi.common.table.timeline.HoodieDefaultTimeline

All Implemented Interfaces:

Serializable, HoodieTimeline

Direct Known Subclasses:

HoodieActiveTimeline, HoodieArchivedTimeline

public class HoodieDefaultTimeline
extends Object
implements HoodieTimeline

HoodieDefaultTimeline is a default implementation of the HoodieTimeline. It provides methods to inspect a List[HoodieInstant]. Function to get the details of the instant is passed in as a lambda.

See Also:

HoodieTimeline, Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected Function<HoodieInstant,Option<byte[]>>	details

Fields inherited from interface org.apache.hudi.common.table.timeline.HoodieTimeline

CLEAN_ACTION, CLEAN_EXTENSION, COMMIT_ACTION, COMMIT_EXTENSION, COMPACTION_ACTION, DELTA_COMMIT_ACTION, DELTA_COMMIT_EXTENSION, EQUALS, FULL_BOOTSTRAP_INSTANT_TS, GREATER_THAN, GREATER_THAN_OR_EQUALS, INDEX_COMMIT_EXTENSION, INDEXING_ACTION, INFLIGHT_CLEAN_EXTENSION, INFLIGHT_COMMIT_EXTENSION, INFLIGHT_COMPACTION_EXTENSION, INFLIGHT_DELTA_COMMIT_EXTENSION, INFLIGHT_EXTENSION, INFLIGHT_INDEX_COMMIT_EXTENSION, INFLIGHT_REPLACE_COMMIT_EXTENSION, INFLIGHT_RESTORE_EXTENSION, INFLIGHT_ROLLBACK_EXTENSION, INFLIGHT_SAVE_SCHEMA_ACTION_EXTENSION, INFLIGHT_SAVEPOINT_EXTENSION, INIT_INSTANT_TS, INVALID_INSTANT_TS, LESSER_THAN, LESSER_THAN_OR_EQUALS, METADATA_BOOTSTRAP_INSTANT_TS, REPLACE_COMMIT_ACTION, REPLACE_COMMIT_EXTENSION, REQUESTED_CLEAN_EXTENSION, REQUESTED_COMMIT_EXTENSION, REQUESTED_COMPACTION_EXTENSION, REQUESTED_COMPACTION_SUFFIX, REQUESTED_DELTA_COMMIT_EXTENSION, REQUESTED_EXTENSION, REQUESTED_INDEX_COMMIT_EXTENSION, REQUESTED_REPLACE_COMMIT_EXTENSION, REQUESTED_RESTORE_EXTENSION, REQUESTED_ROLLBACK_EXTENSION, REQUESTED_SAVE_SCHEMA_ACTION_EXTENSION, RESTORE_ACTION, RESTORE_EXTENSION, ROLLBACK_ACTION, ROLLBACK_EXTENSION, SAVE_SCHEMA_ACTION_EXTENSION, SAVEPOINT_ACTION, SAVEPOINT_EXTENSION, SCHEMA_COMMIT_ACTION, VALID_ACTIONS_IN_TIMELINE

Constructor Summary

Constructors

Constructor and Description
HoodieDefaultTimeline() Deprecated.
HoodieDefaultTimeline(Stream<HoodieInstant> instants, Function<HoodieInstant,Option<byte[]>> details)

Method Summary

Modifier and Type	Method and Description
boolean	containsInstant(HoodieInstant instant)
boolean	containsInstant(String ts)
boolean	containsOrBeforeTimelineStarts(String instant)
int	countInstants()
boolean	empty() If the timeline has any instants.
HoodieTimeline	filter(Predicate<HoodieInstant> filter) Custom Filter of Instants.
HoodieTimeline	filterCompletedAndCompactionInstants() Filter this timeline to just include the completed + compaction (inflight + requested) instants A RT filesystem view is constructed with this timeline so that file-slice after pending compaction-requested instant-time is also considered valid.
HoodieTimeline	filterCompletedIndexTimeline() Filter this timeline to just include completed index instants.
HoodieTimeline	filterCompletedInstants() Filter this timeline to just include the completed instants.
HoodieTimeline	filterInflights() Filter this timeline to just include the in-flights.
HoodieTimeline	filterInflightsAndRequested() Filter this timeline to include requested and in-flights.
protected Stream<HoodieInstant>	filterInstantsByAction(String action)
HoodieTimeline	filterPendingCompactionTimeline() Filter this timeline to just include requested and inflight compaction instants.
HoodieTimeline	filterPendingExcludingCompaction() Filter this timeline to just include the in-flights excluding compaction instants.
HoodieTimeline	filterPendingIndexTimeline() Filter this timeline to just include requested and inflight index instants.
HoodieTimeline	filterPendingReplaceTimeline() Filter this timeline to just include requested and inflight replacecommit instants.
HoodieTimeline	filterPendingRollbackTimeline() Filter this timeline to include pending rollbacks.
HoodieTimeline	findInstantsAfter(String instantTime) Create a new Timeline with all the instants after startTs.
HoodieDefaultTimeline	findInstantsAfter(String instantTime, int numCommits) Create a new Timeline with all the instants after startTs.
HoodieDefaultTimeline	findInstantsAfterOrEquals(String commitTime, int numCommits) Create a new Timeline with all the instants after startTs.
HoodieDefaultTimeline	findInstantsBefore(String instantTime) Create a new Timeline with all instants before specified time.
HoodieDefaultTimeline	findInstantsBeforeOrEquals(String instantTime) Create new timeline with all instants before or equals specified time.
HoodieDefaultTimeline	findInstantsInRange(String startTs, String endTs) Create a new Timeline with instants after startTs and before or on endTs.
Option<HoodieInstant>	firstInstant()
Option<HoodieInstant>	firstInstant(String action, HoodieInstant.State state)
HoodieTimeline	getAllCommitsTimeline() Get all instants (commits, delta commits, compaction, clean, savepoint, rollback, replace commits, index) that result in actions, in the active timeline.
HoodieTimeline	getCleanerTimeline() Get only the cleaner action (inflight and completed) in the active timeline.
HoodieTimeline	getCommitsTimeline() Get all instants (commits, delta commits) that produce new data, in the active timeline.
HoodieTimeline	getCommitTimeline() Get only pure commits (inflight and completed) in the active timeline.
HoodieTimeline	getCompletedReplaceTimeline() Timeline to just include replace instants that have valid (commit/deltacommit) actions.
HoodieTimeline	getContiguousCompletedWriteTimeline() Timeline to just include commits (commit/deltacommit), compaction and replace actions that are completed and contiguous.
HoodieTimeline	getDeltaCommitTimeline() Get only the delta commits (inflight and completed) in the active timeline.
Option<HoodieInstant>	getFirstNonSavepointCommit() First non-savepoint commit in the active data timeline.
Option<byte[]>	getInstantDetails(HoodieInstant instant) Read the completed instant details.
Stream<HoodieInstant>	getInstants()
HoodieTimeline	getRestoreTimeline() Get only the restore action (inflight and completed) in the active timeline.
Stream<HoodieInstant>	getReverseOrderedInstants()
HoodieTimeline	getRollbackAndRestoreTimeline() Get only the rollback and restore action (inflight and completed) in the active timeline.
HoodieTimeline	getRollbackTimeline() Get only the rollback action (inflight and completed) in the active timeline.
HoodieTimeline	getSavePointTimeline() Get only the save point action (inflight and completed) in the active timeline.
String	getTimelineHash() Get hash of timeline.
HoodieTimeline	getTimelineOfActions(Set<String> actions) Get a timeline of a specific set of actions.
HoodieDefaultTimeline	getWriteTimeline() Timeline to just include commits (commit/deltacommit), compaction and replace actions.
boolean	isBeforeTimelineStarts(String instant)
boolean	isEmpty(HoodieInstant instant)
Option<HoodieInstant>	lastInstant()
HoodieDefaultTimeline	mergeTimeline(HoodieDefaultTimeline timeline) Merge this timeline with the given timeline.
Option<HoodieInstant>	nthFromLastInstant(int n)
Option<HoodieInstant>	nthInstant(int n)
void	setInstants(List<HoodieInstant> instants)
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hudi.common.table.timeline.HoodieTimeline

compareTimestamps, getCleanInflightInstant, getCleanRequestedInstant, getCommitFromCommitFile, getCompactionInflightInstant, getCompactionRequestedInstant, getCompletedInstant, getIndexInflightInstant, getIndexRequestedInstant, getInflightInstant, getReplaceCommitInflightInstant, getReplaceCommitRequestedInstant, getRequestedInstant, getRollbackRequestedInstant, isInRange, makeCleanerFileName, makeCommitFileName, makeDeltaFileName, makeFileNameAsComplete, makeFileNameAsInflight, makeIndexCommitFileName, makeInflightCleanerFileName, makeInflightCommitFileName, makeInflightCompactionFileName, makeInflightDeltaFileName, makeInflightIndexFileName, makeInflightReplaceFileName, makeInflightRestoreFileName, makeInflightRollbackFileName, makeInflightSavePointFileName, makeInflightSchemaFileName, makeReplaceFileName, makeRequestedCleanerFileName, makeRequestedCommitFileName, makeRequestedCompactionFileName, makeRequestedDeltaFileName, makeRequestedIndexFileName, makeRequestedReplaceFileName, makeRequestedRestoreFileName, makeRequestedRollbackFileName, makeRequestSchemaFileName, makeRestoreFileName, makeRollbackFileName, makeSavePointFileName, makeSchemaFileName

Field Detail

details

protected transient Function<HoodieInstant,Option<byte[]>> details

Constructor Detail

HoodieDefaultTimeline

public HoodieDefaultTimeline(Stream<HoodieInstant> instants,
                             Function<HoodieInstant,Option<byte[]>> details)

HoodieDefaultTimeline

public HoodieDefaultTimeline()

Deprecated.

For serializing and de-serializing.

Method Detail

setInstants

public void setInstants(List<HoodieInstant> instants)

filterInflights

public HoodieTimeline filterInflights()

Description copied from interface: HoodieTimeline

Filter this timeline to just include the in-flights.

Specified by:

filterInflights in interface HoodieTimeline

Returns:

New instance of HoodieTimeline with just in-flights

filterInflightsAndRequested

public HoodieTimeline filterInflightsAndRequested()

Description copied from interface: HoodieTimeline

Filter this timeline to include requested and in-flights.

Specified by:

filterInflightsAndRequested in interface HoodieTimeline

Returns:

New instance of HoodieTimeline with just in-flights and requested instants

filterPendingExcludingCompaction

public HoodieTimeline filterPendingExcludingCompaction()

Description copied from interface: HoodieTimeline

Filter this timeline to just include the in-flights excluding compaction instants.

Specified by:

filterPendingExcludingCompaction in interface HoodieTimeline

Returns:

New instance of HoodieTimeline with just in-flights excluding compaction instants

filterCompletedInstants

public HoodieTimeline filterCompletedInstants()

Description copied from interface: HoodieTimeline

Filter this timeline to just include the completed instants.

Specified by:

filterCompletedInstants in interface HoodieTimeline

Returns:

New instance of HoodieTimeline with just completed instants

filterCompletedAndCompactionInstants

public HoodieTimeline filterCompletedAndCompactionInstants()

Description copied from interface: HoodieTimeline

Filter this timeline to just include the completed + compaction (inflight + requested) instants A RT filesystem view is constructed with this timeline so that file-slice after pending compaction-requested instant-time is also considered valid. A RT file-system view for reading must then merge the file-slices before and after pending compaction instant so that all delta-commits are read.

Specified by:

filterCompletedAndCompactionInstants in interface HoodieTimeline

Returns:

New instance of HoodieTimeline with just completed instants

getWriteTimeline

public HoodieDefaultTimeline getWriteTimeline()

Description copied from interface: HoodieTimeline

Timeline to just include commits (commit/deltacommit), compaction and replace actions.

Specified by:

getWriteTimeline in interface HoodieTimeline

Returns:

getContiguousCompletedWriteTimeline

public HoodieTimeline getContiguousCompletedWriteTimeline()

Description copied from interface: HoodieTimeline

Timeline to just include commits (commit/deltacommit), compaction and replace actions that are completed and contiguous. For example, if timeline is [C0.completed, C1.completed, C2.completed, C3.inflight, C4.completed]. Then, a timeline of [C0.completed, C1.completed, C2.completed] will be returned.

Specified by:

getContiguousCompletedWriteTimeline in interface HoodieTimeline

Returns:

getCompletedReplaceTimeline

public HoodieTimeline getCompletedReplaceTimeline()

Description copied from interface: HoodieTimeline

Timeline to just include replace instants that have valid (commit/deltacommit) actions.

Specified by:

getCompletedReplaceTimeline in interface HoodieTimeline

Returns:

filterPendingReplaceTimeline

public HoodieTimeline filterPendingReplaceTimeline()

Description copied from interface: HoodieTimeline

Filter this timeline to just include requested and inflight replacecommit instants.

Specified by:

filterPendingReplaceTimeline in interface HoodieTimeline

filterPendingRollbackTimeline

public HoodieTimeline filterPendingRollbackTimeline()

Description copied from interface: HoodieTimeline

Filter this timeline to include pending rollbacks.

Specified by:

filterPendingRollbackTimeline in interface HoodieTimeline

filterPendingCompactionTimeline

public HoodieTimeline filterPendingCompactionTimeline()

Description copied from interface: HoodieTimeline

Filter this timeline to just include requested and inflight compaction instants.

Specified by:

filterPendingCompactionTimeline in interface HoodieTimeline

Returns:

findInstantsInRange

public HoodieDefaultTimeline findInstantsInRange(String startTs,
                                                 String endTs)

Description copied from interface: HoodieTimeline

Create a new Timeline with instants after startTs and before or on endTs.

Specified by:

findInstantsInRange in interface HoodieTimeline

findInstantsAfter

public HoodieDefaultTimeline findInstantsAfter(String instantTime,
                                               int numCommits)

Description copied from interface: HoodieTimeline

Create a new Timeline with all the instants after startTs.

Specified by:

findInstantsAfter in interface HoodieTimeline

findInstantsAfter

public HoodieTimeline findInstantsAfter(String instantTime)

Description copied from interface: HoodieTimeline

Create a new Timeline with all the instants after startTs.

Specified by:

findInstantsAfter in interface HoodieTimeline

findInstantsAfterOrEquals

public HoodieDefaultTimeline findInstantsAfterOrEquals(String commitTime,
                                                       int numCommits)

Description copied from interface: HoodieTimeline

Create a new Timeline with all the instants after startTs.

Specified by:

findInstantsAfterOrEquals in interface HoodieTimeline

findInstantsBefore

public HoodieDefaultTimeline findInstantsBefore(String instantTime)

Description copied from interface: HoodieTimeline

Create a new Timeline with all instants before specified time.

Specified by:

findInstantsBefore in interface HoodieTimeline

findInstantsBeforeOrEquals

public HoodieDefaultTimeline findInstantsBeforeOrEquals(String instantTime)

Description copied from interface: HoodieTimeline

Create new timeline with all instants before or equals specified time.

Specified by:

findInstantsBeforeOrEquals in interface HoodieTimeline

filter

public HoodieTimeline filter(Predicate<HoodieInstant> filter)

Description copied from interface: HoodieTimeline

Custom Filter of Instants.

Specified by:

filter in interface HoodieTimeline

filterPendingIndexTimeline

public HoodieTimeline filterPendingIndexTimeline()

Description copied from interface: HoodieTimeline

Filter this timeline to just include requested and inflight index instants.

Specified by:

filterPendingIndexTimeline in interface HoodieTimeline

filterCompletedIndexTimeline

public HoodieTimeline filterCompletedIndexTimeline()

Description copied from interface: HoodieTimeline

Filter this timeline to just include completed index instants.

Specified by:

filterCompletedIndexTimeline in interface HoodieTimeline

getCommitsTimeline

public HoodieTimeline getCommitsTimeline()

Get all instants (commits, delta commits) that produce new data, in the active timeline.

getAllCommitsTimeline

public HoodieTimeline getAllCommitsTimeline()

Get all instants (commits, delta commits, compaction, clean, savepoint, rollback, replace commits, index) that result in actions, in the active timeline.

getCommitTimeline

public HoodieTimeline getCommitTimeline()

Get only pure commits (inflight and completed) in the active timeline.

getDeltaCommitTimeline

public HoodieTimeline getDeltaCommitTimeline()

Get only the delta commits (inflight and completed) in the active timeline.

getTimelineOfActions

public HoodieTimeline getTimelineOfActions(Set<String> actions)

Get a timeline of a specific set of actions. useful to create a merged timeline of multiple actions.

Parameters:

actions - actions allowed in the timeline

getCleanerTimeline

public HoodieTimeline getCleanerTimeline()

Get only the cleaner action (inflight and completed) in the active timeline.

getRollbackTimeline

public HoodieTimeline getRollbackTimeline()

Get only the rollback action (inflight and completed) in the active timeline.

getRollbackAndRestoreTimeline

public HoodieTimeline getRollbackAndRestoreTimeline()

Get only the rollback and restore action (inflight and completed) in the active timeline.

getSavePointTimeline

public HoodieTimeline getSavePointTimeline()

Get only the save point action (inflight and completed) in the active timeline.

getRestoreTimeline

public HoodieTimeline getRestoreTimeline()

Get only the restore action (inflight and completed) in the active timeline.

filterInstantsByAction

protected Stream<HoodieInstant> filterInstantsByAction(String action)

empty

public boolean empty()

Description copied from interface: HoodieTimeline

If the timeline has any instants.

Specified by:

empty in interface HoodieTimeline

Returns:

true if timeline is empty

countInstants

public int countInstants()

Specified by:

countInstants in interface HoodieTimeline

Returns:

total number of completed instants

firstInstant

public Option<HoodieInstant> firstInstant()

Specified by:

firstInstant in interface HoodieTimeline

Returns:

first completed instant if available

firstInstant

public Option<HoodieInstant> firstInstant(String action,
                                          HoodieInstant.State state)

Specified by:

firstInstant in interface HoodieTimeline

Parameters:

action - Instant action String.

state - Instant State.

Returns:

first instant of a specific action and state if available

nthInstant

public Option<HoodieInstant> nthInstant(int n)

Specified by:

nthInstant in interface HoodieTimeline

Returns:

nth completed instant from the first completed instant

lastInstant

public Option<HoodieInstant> lastInstant()

Specified by:

lastInstant in interface HoodieTimeline

Returns:

last completed instant if available

nthFromLastInstant

public Option<HoodieInstant> nthFromLastInstant(int n)

Specified by:

nthFromLastInstant in interface HoodieTimeline

Returns:

nth completed instant going back from the last completed instant

containsInstant

public boolean containsInstant(HoodieInstant instant)

Specified by:

containsInstant in interface HoodieTimeline

Returns:

true if the passed instant is present as a completed instant on the timeline

containsInstant

public boolean containsInstant(String ts)

Specified by:

containsInstant in interface HoodieTimeline

Returns:

true if the passed instant is present as a completed instant on the timeline

containsOrBeforeTimelineStarts

public boolean containsOrBeforeTimelineStarts(String instant)

Specified by:

containsOrBeforeTimelineStarts in interface HoodieTimeline

Returns:

true if the passed instant is present as a completed instant on the timeline or if the instant is before the first completed instant in the timeline

getTimelineHash

public String getTimelineHash()

Description copied from interface: HoodieTimeline

Get hash of timeline.

Specified by:

getTimelineHash in interface HoodieTimeline

Returns:

getInstants

public Stream<HoodieInstant> getInstants()

Specified by:

getInstants in interface HoodieTimeline

Returns:

Get the stream of completed instants

getReverseOrderedInstants

public Stream<HoodieInstant> getReverseOrderedInstants()

Specified by:

getReverseOrderedInstants in interface HoodieTimeline

Returns:

Get the stream of completed instants in reverse order TODO Change code references to getInstants() that reverse the instants later on to use this method instead.

isBeforeTimelineStarts

public boolean isBeforeTimelineStarts(String instant)

Specified by:

isBeforeTimelineStarts in interface HoodieTimeline

Returns:

true if the passed in instant is before the first completed instant in the timeline

getFirstNonSavepointCommit

public Option<HoodieInstant> getFirstNonSavepointCommit()

Description copied from interface: HoodieTimeline

First non-savepoint commit in the active data timeline. Examples: 1. An active data timeline C1, C2, C3, C4, C5 returns C1. 2. If archival is allowed beyond savepoint and let's say C1, C2, C4 have been archived while C3, C5 have been savepointed, then for the data timeline C3, C3_Savepoint, C5, C5_Savepoint, C6, C7 returns C6.

Specified by:

getFirstNonSavepointCommit in interface HoodieTimeline

getInstantDetails

public Option<byte[]> getInstantDetails(HoodieInstant instant)

Description copied from interface: HoodieTimeline

Read the completed instant details.

Specified by:

getInstantDetails in interface HoodieTimeline

isEmpty

public boolean isEmpty(HoodieInstant instant)

Specified by:

isEmpty in interface HoodieTimeline

toString

public String toString()

Overrides:

toString in class Object

mergeTimeline

public HoodieDefaultTimeline mergeTimeline(HoodieDefaultTimeline timeline)

Merge this timeline with the given timeline.