org.apache.druid.segment.loading

Interface DataSegmentFinder

public interface DataSegmentFinder

A DataSegmentFinder is responsible for finding Druid segments underneath a specified directory and optionally updates all descriptor.json files on deep storage with correct loadSpec.

Field Summary

Fields

Modifier and Type	Field and Description
static Logger	log

Method Summary

Modifier and Type	Method and Description
Set<DataSegment>	findSegments(String workingDirPath, boolean updateDescriptor) This method should first recursively look for descriptor.json (partitionNum_descriptor.json for HDFS data storage) underneath workingDirPath and then verify that index.zip (partitionNum_index.zip for HDFS data storage) exists in the same folder.
static void	putInMapRetainingNewest(Map<String,Pair<DataSegment,Long>> timestampedSegments, DataSegment dataSegment, long segmentModifiedAt) Adds dataSegment if it does not exist in timestampedSegments.

Field Detail

log

static final Logger log

Method Detail

findSegments

Set<DataSegment> findSegments(String workingDirPath,
                              boolean updateDescriptor)
                       throws SegmentLoadingException

This method should first recursively look for descriptor.json (partitionNum_descriptor.json for HDFS data storage) underneath workingDirPath and then verify that index.zip (partitionNum_index.zip for HDFS data storage) exists in the same folder. If not, it should throw SegmentLoadingException to let the caller know that descriptor.json exists while index.zip doesn't. If a segment is found and updateDescriptor is set, then this method should update the loadSpec in descriptor.json to reflect the location from where it was found. After the search, this method should return the set of segments that were found.

Parameters:

workingDirPath - the String representation of the working directory path

updateDescriptor - if true, update loadSpec in descriptor.json if loadSpec's location is different from where desciptor.json was found

Returns:

a set of segments that were found underneath workingDirPath

Throws:

SegmentLoadingException

putInMapRetainingNewest

static void putInMapRetainingNewest(Map<String,Pair<DataSegment,Long>> timestampedSegments,
                                    DataSegment dataSegment,
                                    long segmentModifiedAt)

Adds dataSegment if it does not exist in timestampedSegments. If it exists, replaces entry if segmentModifiedAt is newer than stored timestamp.

Parameters:

timestampedSegments - map of > containing segments with modified time

dataSegment - segment to add

segmentModifiedAt - segment modified timestamp