Package org.apache.druid.server.coordinator.duty

Class MarkEternityTombstonesAsUnused

java.lang.Object

org.apache.druid.server.coordinator.duty.MarkEternityTombstonesAsUnused

All Implemented Interfaces:

CoordinatorDuty

public class MarkEternityTombstonesAsUnused
extends Object
implements CoordinatorDuty

Mark eternity tombstones not overshadowed by currently served segments as unused. A candidate segment must fit all the criteria:

It is a tombstone that starts at DateTimes.MIN or ends at DateTimes.MAX It does not overlap with any overshadowed segment in the datasource It has has 0 core partitions i.e., TombstoneShardSpec.getNumCorePartitions() == 0

Only infinite-interval tombstones are considered as candidate segments in this duty because they don't honor the preferred segment granularity specified at ingest time to cover an underlying segment with Granularities.ALL as it can generate too many segments per time chunk and cause an OOM. The infinite-interval tombstones make it hard to append data on the end of a data set that started out with an Granularities.ALL eternity and then moved to actual time grains, so the compromise is that the coordinator will remove these segments as long as it doesn't overlap any other segment.

The overlapping condition is necessary as a candidate segment can overlap with an overshadowed segment, and the latter needs to be marked as unused first by MarkOvershadowedSegmentsAsUnused duty before the tombstone candidate can be marked as unused by MarkEternityTombstonesAsUnused duty.

Only tombstones with 0 core partitions is considered as candidate segments. Earlier generation tombstones with 1 core partition (i.e., TombstoneShardSpec.getNumCorePartitions() == 1) are ignored by this duty because it can potentially cause data loss in a concurrent append and replace scenario and needs to be manually cleaned up. See this for details.

Constructor Summary

Constructors

Constructor	Description
MarkEternityTombstonesAsUnused(SegmentDeleteHandler deleteHandler)

Method Summary

Modifier and Type	Method	Description
DruidCoordinatorRuntimeParams	run(DruidCoordinatorRuntimeParams params)	Implementations of this method run various activities performed by the coordinator.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MarkEternityTombstonesAsUnused

public MarkEternityTombstonesAsUnused(SegmentDeleteHandler deleteHandler)

Method Detail

run

public DruidCoordinatorRuntimeParams run(DruidCoordinatorRuntimeParams params)

Description copied from interface: CoordinatorDuty

Implementations of this method run various activities performed by the coordinator. Input params can be used and modified. The returned DruidCoordinatorRuntimeParams is passed to the next duty.

Specified by:

run in interface CoordinatorDuty

Returns:

same as input or a modified value to be used by next duty. Null return values will prevent subsequent CoordinatorDuty objects (see the order in the lists passed into the constructor of DruidCoordinator.DutiesRunnable) from running until the next cycle.