Package org.apache.pinot.segment.local.utils

Class SegmentPushUtils

java.lang.Object

org.apache.pinot.segment.local.utils.SegmentPushUtils

All Implemented Interfaces:

Serializable

public class SegmentPushUtils
extends Object
implements Serializable

See Also:

Serialized Form

Method Summary

Modifier and Type	Method	Description
static URI	generateSegmentTarURI(URI dirURI, URI fileURI, String prefix, String suffix)
static Map<String,String>	getSegmentUriToTarPathMap(URI outputDirURI, PushJobSpec pushSpec, String[] files)
static void	pushSegments(SegmentGenerationJobSpec spec, PinotFS fileSystem, List<String> tarFilePaths)
static void	pushSegments(SegmentGenerationJobSpec spec, PinotFS fileSystem, List<String> tarFilePaths, List<org.apache.http.Header> headers, List<org.apache.http.NameValuePair> parameters)
static void	sendSegmentUriAndMetadata(SegmentGenerationJobSpec spec, PinotFS fileSystem, Map<String,String> segmentUriToTarPathMap)	This method takes a map of segment downloadURI to corresponding tar file path, and push those segments in metadata mode.
static void	sendSegmentUriAndMetadata(SegmentGenerationJobSpec spec, PinotFS fileSystem, Map<String,String> segmentUriToTarPathMap, List<org.apache.http.Header> headers, List<org.apache.http.NameValuePair> parameters)	This method takes a map of segment downloadURI to corresponding tar file path, and push those segments in metadata mode.
static void	sendSegmentUris(SegmentGenerationJobSpec spec, List<String> segmentUris)
static void	sendSegmentUris(SegmentGenerationJobSpec spec, List<String> segmentUris, List<org.apache.http.Header> headers, List<org.apache.http.NameValuePair> parameters)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

generateSegmentTarURI

public static URI generateSegmentTarURI(URI dirURI,
                                        URI fileURI,
                                        String prefix,
                                        String suffix)

pushSegments

public static void pushSegments(SegmentGenerationJobSpec spec,
                                PinotFS fileSystem,
                                List<String> tarFilePaths)
                         throws RetriableOperationException,
                                AttemptsExceededException

Throws:

RetriableOperationException

AttemptsExceededException

sendSegmentUris

public static void sendSegmentUris(SegmentGenerationJobSpec spec,
                                   List<String> segmentUris)
                            throws RetriableOperationException,
                                   AttemptsExceededException

Throws:

RetriableOperationException

AttemptsExceededException

sendSegmentUriAndMetadata

public static void sendSegmentUriAndMetadata(SegmentGenerationJobSpec spec,
                                             PinotFS fileSystem,
                                             Map<String,String> segmentUriToTarPathMap)
                                      throws Exception

This method takes a map of segment downloadURI to corresponding tar file path, and push those segments in metadata mode. The steps are: 1. Download segment from tar file path; 2. Untar segment metadata and creation meta files from the tar file to a segment metadata directory; 3. Tar this segment metadata directory into a tar file 4. Generate a POST request with segmentDownloadURI in header to push tar file to Pinot controller.

Parameters:

spec - is the segment generation job spec

fileSystem - is the PinotFs used to copy segment tar file

segmentUriToTarPathMap - contains the map of segment DownloadURI to segment tar file path

Throws:

Exception

pushSegments

public static void pushSegments(SegmentGenerationJobSpec spec,
                                PinotFS fileSystem,
                                List<String> tarFilePaths,
                                List<org.apache.http.Header> headers,
                                List<org.apache.http.NameValuePair> parameters)
                         throws RetriableOperationException,
                                AttemptsExceededException

Throws:

RetriableOperationException

AttemptsExceededException

sendSegmentUris

public static void sendSegmentUris(SegmentGenerationJobSpec spec,
                                   List<String> segmentUris,
                                   List<org.apache.http.Header> headers,
                                   List<org.apache.http.NameValuePair> parameters)
                            throws RetriableOperationException,
                                   AttemptsExceededException

Throws:

RetriableOperationException

AttemptsExceededException

sendSegmentUriAndMetadata

public static void sendSegmentUriAndMetadata(SegmentGenerationJobSpec spec,
                                             PinotFS fileSystem,
                                             Map<String,String> segmentUriToTarPathMap,
                                             List<org.apache.http.Header> headers,
                                             List<org.apache.http.NameValuePair> parameters)
                                      throws Exception

This method takes a map of segment downloadURI to corresponding tar file path, and push those segments in metadata mode. The steps are: 1. Download segment from tar file path; 2. Untar segment metadata and creation meta files from the tar file to a segment metadata directory; 3. Tar this segment metadata directory into a tar file 4. Generate a POST request with segmentDownloadURI in header to push tar file to Pinot controller.

Parameters:

spec - is the segment generation job spec

fileSystem - is the PinotFs used to copy segment tar file

segmentUriToTarPathMap - contains the map of segment DownloadURI to segment tar file path

Throws:

Exception

getSegmentUriToTarPathMap

public static Map<String,String> getSegmentUriToTarPathMap(URI outputDirURI,
                                                                 PushJobSpec pushSpec,
                                                                 String[] files)