org.apache.druid.indexing.overlord

Interface IndexerMetadataStorageCoordinator

All Known Implementing Classes:

IndexerSQLMetadataStorageCoordinator

public interface IndexerMetadataStorageCoordinator

Method Summary

Modifier and Type	Method and Description
SegmentIdWithShardSpec	allocatePendingSegment(String dataSource, String sequenceName, String previousSegmentId, org.joda.time.Interval interval, PartialShardSpec partialShardSpec, String maxVersion, boolean skipSegmentLineageCheck) Allocate a new pending segment in the pending segments table.
Set<DataSegment>	announceHistoricalSegments(Set<DataSegment> segments) Attempts to insert a set of segments to the metadata storage.
SegmentPublishResult	announceHistoricalSegments(Set<DataSegment> segments, DataSourceMetadata startMetadata, DataSourceMetadata endMetadata) Attempts to insert a set of segments to the metadata storage.
SegmentPublishResult	commitMetadataOnly(String dataSource, DataSourceMetadata startMetadata, DataSourceMetadata endMetadata) Similar to announceHistoricalSegments(Set), but meant for streaming ingestion tasks for handling the case where the task ingested no records and created no segments, but still needs to update the metadata with the progress that the task made.
boolean	deleteDataSourceMetadata(String dataSource) Removes entry for 'dataSource' from the dataSource metadata table.
int	deletePendingSegments(String dataSource) Delete all pending segments belonging to the given data source from the pending segments table.
int	deletePendingSegmentsCreatedInInterval(String dataSource, org.joda.time.Interval deleteInterval) Delete pending segments created in the given interval belonging to the given data source from the pending segments table.
void	deleteSegments(Set<DataSegment> segments)
boolean	insertDataSourceMetadata(String dataSource, DataSourceMetadata dataSourceMetadata) Insert dataSourceMetadata entry for 'dataSource'.
boolean	resetDataSourceMetadata(String dataSource, DataSourceMetadata dataSourceMetadata) Resets dataSourceMetadata entry for 'dataSource' to the one supplied.
Collection<DataSegment>	retrieveAllUsedSegments(String dataSource, Segments visibility) Retrieve all published used segments in the data source from the metadata store.
DataSourceMetadata	retrieveDataSourceMetadata(String dataSource) Retrieves data source's metadata from the metadata store.
List<DataSegment>	retrieveUnusedSegmentsForInterval(String dataSource, org.joda.time.Interval interval) Retrieve all published segments which include ONLY data within the given interval and are marked as unused from the metadata store.
Collection<Pair<DataSegment,String>>	retrieveUsedSegmentsAndCreatedDates(String dataSource) Retrieve all published segments which are marked as used and the created_date of these segments belonging to the given data source from the metadata store.
default Collection<DataSegment>	retrieveUsedSegmentsForInterval(String dataSource, org.joda.time.Interval interval, Segments visibility) Retrieve all published segments which may include any data in the interval and are marked as used from the metadata store.
Collection<DataSegment>	retrieveUsedSegmentsForIntervals(String dataSource, List<org.joda.time.Interval> intervals, Segments visibility) Retrieve all published segments which may include any data in the given intervals and are marked as used from the metadata store.
void	updateSegmentMetadata(Set<DataSegment> segments)

Method Detail

retrieveUsedSegmentsForInterval

default Collection<DataSegment> retrieveUsedSegmentsForInterval(String dataSource,
                                                                org.joda.time.Interval interval,
                                                                Segments visibility)

Retrieve all published segments which may include any data in the interval and are marked as used from the metadata store. The order of segments within the returned collection is unspecified, but each segment is guaranteed to appear in the collection only once.

Parameters:

dataSource - The data source to query

interval - The interval for which all applicable and used segmented are requested.

visibility - Whether only visible or visible as well as overshadowed segments should be returned. The visibility is considered within the specified interval: that is, a segment which is visible outside of the specified interval, but overshadowed within the specified interval will not be returned if Segments.ONLY_VISIBLE is passed. See more precise description in the doc for Segments.

Returns:

The DataSegments which include data in the requested interval. These segments may contain data outside the requested interval.

retrieveAllUsedSegments

Collection<DataSegment> retrieveAllUsedSegments(String dataSource,
                                                Segments visibility)

Retrieve all published used segments in the data source from the metadata store.

Parameters:

dataSource - The data source to query

Returns:

all segments belonging to the given data source

See Also:

similar to this method but also accepts data interval.

retrieveUsedSegmentsAndCreatedDates

Collection<Pair<DataSegment,String>> retrieveUsedSegmentsAndCreatedDates(String dataSource)

Retrieve all published segments which are marked as used and the created_date of these segments belonging to the given data source from the metadata store. Unlike other similar methods in this interface, this method doesn't accept a Segments "visibility" parameter. The returned collection may include overshadowed segments and their created_dates, as if Segments.INCLUDING_OVERSHADOWED was passed. It's the responsibility of the caller to filter out overshadowed ones if needed.

Parameters:

dataSource - The data source to query

Returns:

The DataSegments and the related created_date of segments

retrieveUsedSegmentsForIntervals

Collection<DataSegment> retrieveUsedSegmentsForIntervals(String dataSource,
                                                         List<org.joda.time.Interval> intervals,
                                                         Segments visibility)

Retrieve all published segments which may include any data in the given intervals and are marked as used from the metadata store. The order of segments within the returned collection is unspecified, but each segment is guaranteed to appear in the collection only once.

Parameters:

dataSource - The data source to query

intervals - The intervals for which all applicable and used segments are requested.

visibility - Whether only visible or visible as well as overshadowed segments should be returned. The visibility is considered within the specified intervals: that is, a segment which is visible outside of the specified intervals, but overshadowed on the specified intervals will not be returned if Segments.ONLY_VISIBLE is passed. See more precise description in the doc for Segments.

Returns:

The DataSegments which include data in the requested intervals. These segments may contain data outside the requested intervals.

retrieveUnusedSegmentsForInterval

List<DataSegment> retrieveUnusedSegmentsForInterval(String dataSource,
                                                    org.joda.time.Interval interval)

Retrieve all published segments which include ONLY data within the given interval and are marked as unused from the metadata store.

Parameters:

dataSource - The data source the segments belong to

interval - Filter the data segments to ones that include data in this interval exclusively.

Returns:

DataSegments which include ONLY data within the requested interval and are marked as unused. Segments NOT returned here may include data in the interval

announceHistoricalSegments

Set<DataSegment> announceHistoricalSegments(Set<DataSegment> segments)
                                     throws IOException

Attempts to insert a set of segments to the metadata storage. Returns the set of segments actually added (segments with identifiers already in the metadata storage will not be added).

Parameters:

segments - set of segments to add

Returns:

set of segments actually added

Throws:

IOException

allocatePendingSegment

SegmentIdWithShardSpec allocatePendingSegment(String dataSource,
                                              String sequenceName,
                                              @Nullable
                                              String previousSegmentId,
                                              org.joda.time.Interval interval,
                                              PartialShardSpec partialShardSpec,
                                              String maxVersion,
                                              boolean skipSegmentLineageCheck)

Allocate a new pending segment in the pending segments table. This segment identifier will never be given out again, unless another call is made with the same dataSource, sequenceName, and previousSegmentId.

The sequenceName and previousSegmentId parameters are meant to make it easy for two independent ingestion tasks to produce the same series of segments.

Note that a segment sequence may include segments with a variety of different intervals and versions.

Parameters:

dataSource - dataSource for which to allocate a segment

sequenceName - name of the group of ingestion tasks producing a segment series

previousSegmentId - previous segment in the series; may be null or empty, meaning this is the first segment

interval - interval for which to allocate a segment

partialShardSpec - partialShardSpec containing all necessary information to create a shardSpec for the new segmentId

maxVersion - use this version if we have no better version to use. The returned segment identifier may have a version lower than this one, but will not have one higher.

skipSegmentLineageCheck - if true, perform lineage validation using previousSegmentId for this sequence. Should be set to false if replica tasks would index events in same order

Returns:

the pending segment identifier, or null if it was impossible to allocate a new segment

deletePendingSegmentsCreatedInInterval

int deletePendingSegmentsCreatedInInterval(String dataSource,
                                           org.joda.time.Interval deleteInterval)

Delete pending segments created in the given interval belonging to the given data source from the pending segments table. The created_date field of the pending segments table is checked to find segments to be deleted. Note that the semantic of the interval (for `created_date`s) is different from the semantic of the interval parameters in some other methods in this class, such as retrieveUsedSegmentsForInterval(java.lang.String, org.joda.time.Interval, org.apache.druid.indexing.overlord.Segments) (where the interval is about the time column value in rows belonging to the segment).

Parameters:

dataSource - dataSource

deleteInterval - interval to check the created_date of pendingSegments

Returns:

number of deleted pending segments

deletePendingSegments

int deletePendingSegments(String dataSource)

Delete all pending segments belonging to the given data source from the pending segments table.

Returns:

number of deleted pending segments

See Also:

similar to this method but also accepts interval for segments' `created_date`s

announceHistoricalSegments

SegmentPublishResult announceHistoricalSegments(Set<DataSegment> segments,
                                                @Nullable
                                                DataSourceMetadata startMetadata,
                                                @Nullable
                                                DataSourceMetadata endMetadata)
                                         throws IOException

Attempts to insert a set of segments to the metadata storage. Returns the set of segments actually added (segments with identifiers already in the metadata storage will not be added).

If startMetadata and endMetadata are set, this insertion will be atomic with a compare-and-swap on dataSource commit metadata.

Parameters:

segments - set of segments to add, must all be from the same dataSource

startMetadata - dataSource metadata pre-insert must match this startMetadata according to DataSourceMetadata.matches(DataSourceMetadata). If null, this insert will not involve a metadata transaction

endMetadata - dataSource metadata post-insert will have this endMetadata merged in with DataSourceMetadata.plus(DataSourceMetadata). If null, this insert will not involve a metadata transaction

Returns:

segment publish result indicating transaction success or failure, and set of segments actually published. This method must only return a failure code if it is sure that the transaction did not happen. If it is not sure, it must throw an exception instead.

Throws:

IllegalArgumentException - if startMetadata and endMetadata are not either both null or both non-null

RuntimeException - if the state of metadata storage after this call is unknown

IOException

retrieveDataSourceMetadata

@Nullable
DataSourceMetadata retrieveDataSourceMetadata(String dataSource)

Retrieves data source's metadata from the metadata store. Returns null if there is no metadata.

deleteDataSourceMetadata

boolean deleteDataSourceMetadata(String dataSource)

Removes entry for 'dataSource' from the dataSource metadata table.

Parameters:

dataSource - identifier

Returns:

true if the entry was deleted, false otherwise

resetDataSourceMetadata

boolean resetDataSourceMetadata(String dataSource,
                                DataSourceMetadata dataSourceMetadata)
                         throws IOException

Resets dataSourceMetadata entry for 'dataSource' to the one supplied.

Parameters:

dataSource - identifier

dataSourceMetadata - value to set

Returns:

true if the entry was reset, false otherwise

Throws:

IOException

insertDataSourceMetadata

boolean insertDataSourceMetadata(String dataSource,
                                 DataSourceMetadata dataSourceMetadata)

Insert dataSourceMetadata entry for 'dataSource'.

Parameters:

dataSource - identifier

dataSourceMetadata - value to set

Returns:

true if the entry was inserted, false otherwise

commitMetadataOnly

SegmentPublishResult commitMetadataOnly(String dataSource,
                                        DataSourceMetadata startMetadata,
                                        DataSourceMetadata endMetadata)

Similar to announceHistoricalSegments(Set), but meant for streaming ingestion tasks for handling the case where the task ingested no records and created no segments, but still needs to update the metadata with the progress that the task made. The metadata should undergo the same validation checks as performed by announceHistoricalSegments(java.util.Set<org.apache.druid.timeline.DataSegment>).

Parameters:

dataSource - the datasource

startMetadata - dataSource metadata pre-insert must match this startMetadata according to DataSourceMetadata.matches(DataSourceMetadata).

endMetadata - dataSource metadata post-insert will have this endMetadata merged in with DataSourceMetadata.plus(DataSourceMetadata).

Returns:

segment publish result indicating transaction success or failure. This method must only return a failure code if it is sure that the transaction did not happen. If it is not sure, it must throw an exception instead.

Throws:

IllegalArgumentException - if either startMetadata and endMetadata are null

RuntimeException - if the state of metadata storage after this call is unknown

updateSegmentMetadata

void updateSegmentMetadata(Set<DataSegment> segments)

deleteSegments

void deleteSegments(Set<DataSegment> segments)