Package org.apache.pinot.segment.local.utils

Class SegmentPushUtils

java.lang.Object

org.apache.pinot.segment.local.utils.SegmentPushUtils

All Implemented Interfaces:

Serializable

public class SegmentPushUtils
extends Object
implements Serializable

See Also:

Serialized Form

Method Summary

Modifier and Type	Method	Description
static File	generateSegmentMetadataFile(PinotFS fileSystem, URI tarFileURI)	Generate a segment metadata only tar file, which contains only metadata.properties and creation.meta file.
static URI	generateSegmentMetadataURI(String segmentTarPath, String segmentName)
static URI	generateSegmentTarURI(URI dirURI, URI fileURI, String prefix, String suffix)
static Map<String,String>	getSegmentUriToTarPathMap(URI outputDirURI, PushJobSpec pushSpec, String[] files)
static void	pushSegments(SegmentGenerationJobSpec spec, PinotFS fileSystem, List<String> tarFilePaths)
static void	pushSegments(SegmentGenerationJobSpec spec, PinotFS fileSystem, List<String> tarFilePaths, List<org.apache.http.Header> headers, List<org.apache.http.NameValuePair> parameters)
static void	sendSegmentUriAndMetadata(SegmentGenerationJobSpec spec, PinotFS fileSystem, Map<String,String> segmentUriToTarPathMap)	This method takes a map of segment downloadURI to corresponding tar file path, and push those segments in metadata mode.
static void	sendSegmentUriAndMetadata(SegmentGenerationJobSpec spec, PinotFS fileSystem, Map<String,String> segmentUriToTarPathMap, List<org.apache.http.Header> headers, List<org.apache.http.NameValuePair> parameters)	This method takes a map of segment downloadURI to corresponding tar file path, and push those segments in metadata mode.
static void	sendSegmentUris(SegmentGenerationJobSpec spec, List<String> segmentUris)
static void	sendSegmentUris(SegmentGenerationJobSpec spec, List<String> segmentUris, List<org.apache.http.Header> headers, List<org.apache.http.NameValuePair> parameters)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

generateSegmentTarURI

public static URI generateSegmentTarURI(URI dirURI,
                                        URI fileURI,
                                        String prefix,
                                        String suffix)

pushSegments

public static void pushSegments(SegmentGenerationJobSpec spec,
                                PinotFS fileSystem,
                                List<String> tarFilePaths)
                         throws RetriableOperationException,
                                AttemptsExceededException

Throws:

RetriableOperationException

AttemptsExceededException

sendSegmentUris

public static void sendSegmentUris(SegmentGenerationJobSpec spec,
                                   List<String> segmentUris)
                            throws RetriableOperationException,
                                   AttemptsExceededException

Throws:

RetriableOperationException

AttemptsExceededException

sendSegmentUriAndMetadata

public static void sendSegmentUriAndMetadata(SegmentGenerationJobSpec spec,
                                             PinotFS fileSystem,
                                             Map<String,String> segmentUriToTarPathMap)
                                      throws Exception

This method takes a map of segment downloadURI to corresponding tar file path, and push those segments in metadata mode. The steps are: 1. Download segment from tar file path; 2. Untar segment metadata and creation meta files from the tar file to a segment metadata directory; 3. Tar this segment metadata directory into a tar file 4. Generate a POST request with segmentDownloadURI in header to push tar file to Pinot controller.

Parameters:

spec - is the segment generation job spec

fileSystem - is the PinotFs used to copy segment tar file

segmentUriToTarPathMap - contains the map of segment DownloadURI to segment tar file path

Throws:

Exception

pushSegments

public static void pushSegments(SegmentGenerationJobSpec spec,
                                PinotFS fileSystem,
                                List<String> tarFilePaths,
                                List<org.apache.http.Header> headers,
                                List<org.apache.http.NameValuePair> parameters)
                         throws RetriableOperationException,
                                AttemptsExceededException

Throws:

RetriableOperationException

AttemptsExceededException

sendSegmentUris

public static void sendSegmentUris(SegmentGenerationJobSpec spec,
                                   List<String> segmentUris,
                                   List<org.apache.http.Header> headers,
                                   List<org.apache.http.NameValuePair> parameters)
                            throws RetriableOperationException,
                                   AttemptsExceededException

Throws:

RetriableOperationException

AttemptsExceededException

sendSegmentUriAndMetadata

public static void sendSegmentUriAndMetadata(SegmentGenerationJobSpec spec,
                                             PinotFS fileSystem,
                                             Map<String,String> segmentUriToTarPathMap,
                                             List<org.apache.http.Header> headers,
                                             List<org.apache.http.NameValuePair> parameters)
                                      throws Exception

This method takes a map of segment downloadURI to corresponding tar file path, and push those segments in metadata mode. The steps are: 1. Download segment from tar file path; 2. Untar segment metadata and creation meta files from the tar file to a segment metadata directory; 3. Tar this segment metadata directory into a tar file 4. Generate a POST request with segmentDownloadURI in header to push tar file to Pinot controller.

Parameters:

spec - is the segment generation job spec

fileSystem - is the PinotFs used to copy segment tar file

segmentUriToTarPathMap - contains the map of segment DownloadURI to segment tar file path

Throws:

Exception

getSegmentUriToTarPathMap

public static Map<String,String> getSegmentUriToTarPathMap(URI outputDirURI,
                                                                 PushJobSpec pushSpec,
                                                                 String[] files)

generateSegmentMetadataFile

public static File generateSegmentMetadataFile(PinotFS fileSystem,
                                               URI tarFileURI)
                                        throws Exception

Generate a segment metadata only tar file, which contains only metadata.properties and creation.meta file. The purpose of this is to create a lean tar to push to Pinot controller for adding segments without downloading the complete segment and untar the segment tarball. 1. Download segment tar file to temp dir; 2. Extract only metadata.properties and creation.meta files from the segment tar file; 3. Tar both files into a segment metadata file.

Throws:

Exception

generateSegmentMetadataURI

public static URI generateSegmentMetadataURI(String segmentTarPath,
                                             String segmentName)
                                      throws URISyntaxException

Throws:

URISyntaxException