Package org.apache.pinot.segment.local.utils

Class ConsistentDataPushUtils

java.lang.Object

org.apache.pinot.segment.local.utils.ConsistentDataPushUtils

public class ConsistentDataPushUtils
extends Object

Field Summary

Fields

Modifier and Type	Field	Description
static String	SEGMENT_NAME_POSTFIX

Method Summary

Modifier and Type	Method	Description
static void	configureSegmentPostfix(SegmentGenerationJobSpec spec)	Append current timestamp to existing configured segment name postfix, if configured, to make segment name unique.
static boolean	consistentDataPushEnabled(TableConfig tableConfig)
static void	endReplaceSegments(SegmentGenerationJobSpec spec, Map<URI,String> uriToLineageEntryIdMap)	Ends consistent data push protocol for each Pinot cluster in the spec.
static Map<URI,List<String>>	getSegmentsToReplace(SegmentGenerationJobSpec spec, String rawTableName)	Returns a map of controller URI to a list of existing OFFLINE segments.
static Map<URI,URI>	getStartReplaceSegmentUris(SegmentGenerationJobSpec spec, String rawTableName)	Builds a map of controller URI to startReplaceSegments URI for each Pinot cluster in the spec.
static void	handleUploadException(SegmentGenerationJobSpec spec, Map<URI,String> uriToLineageEntryIdMap, Exception exception)	Revert segment lineage entry when exception gets caught.
static void	postUpload(SegmentGenerationJobSpec spec, Map<URI,String> uriToLineageEntryIdMap)	uriToLineageEntryIdMap is non-empty if and only if consistent data push is enabled.
static Map<URI,String>	preUpload(SegmentGenerationJobSpec spec, List<String> segmentsTo)	Fetch the list of segments to be replaced, then invoke startReplaceSegments API and returns a map of controller URI to lineage entry IDs.
static Map<URI,String>	startReplaceSegments(SegmentGenerationJobSpec spec, Map<URI,List<String>> uriToSegmentsFrom, List<String> segmentsTo)	Starts consistent data push protocol for each Pinot cluster in the spec.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SEGMENT_NAME_POSTFIX

public static final String SEGMENT_NAME_POSTFIX

See Also:

Constant Field Values

Method Detail

preUpload

public static Map<URI,String> preUpload(SegmentGenerationJobSpec spec,
                                              List<String> segmentsTo)
                                       throws Exception

Fetch the list of segments to be replaced, then invoke startReplaceSegments API and returns a map of controller URI to lineage entry IDs.

Throws:

Exception

postUpload

public static void postUpload(SegmentGenerationJobSpec spec,
                              Map<URI,String> uriToLineageEntryIdMap)
                       throws Exception

uriToLineageEntryIdMap is non-empty if and only if consistent data push is enabled. If uriToLineageEntryIdMap is non-empty, end the consistent data push protocol for each controller.

Throws:

Exception

getStartReplaceSegmentUris

public static Map<URI,URI> getStartReplaceSegmentUris(SegmentGenerationJobSpec spec,
                                                            String rawTableName)

Builds a map of controller URI to startReplaceSegments URI for each Pinot cluster in the spec.

startReplaceSegments

public static Map<URI,String> startReplaceSegments(SegmentGenerationJobSpec spec,
                                                         Map<URI,List<String>> uriToSegmentsFrom,
                                                         List<String> segmentsTo)
                                                  throws Exception

Starts consistent data push protocol for each Pinot cluster in the spec. Returns a map of controller URI to segment lineage entry ID.

Throws:

Exception

endReplaceSegments

public static void endReplaceSegments(SegmentGenerationJobSpec spec,
                                      Map<URI,String> uriToLineageEntryIdMap)
                               throws Exception

Ends consistent data push protocol for each Pinot cluster in the spec.

Throws:

Exception

handleUploadException

public static void handleUploadException(SegmentGenerationJobSpec spec,
                                         Map<URI,String> uriToLineageEntryIdMap,
                                         Exception exception)

Revert segment lineage entry when exception gets caught. This revert request is called at best effort. If the revert call fails at this point, the next startReplaceSegment call will do the cleanup by marking the previous entry to "REVERTED" and cleaning up the leftover segments.

consistentDataPushEnabled

public static boolean consistentDataPushEnabled(TableConfig tableConfig)

getSegmentsToReplace

public static Map<URI,List<String>> getSegmentsToReplace(SegmentGenerationJobSpec spec,
                                                               String rawTableName)
                                                        throws Exception

Returns a map of controller URI to a list of existing OFFLINE segments.

Throws:

Exception

configureSegmentPostfix

public static void configureSegmentPostfix(SegmentGenerationJobSpec spec)

Append current timestamp to existing configured segment name postfix, if configured, to make segment name unique.