Class SegmentPushUtils
- java.lang.Object
-
- org.apache.pinot.segment.local.utils.SegmentPushUtils
-
- All Implemented Interfaces:
Serializable
public class SegmentPushUtils extends Object implements Serializable
- See Also:
- Serialized Form
-
-
Method Summary
All Methods Static Methods Concrete Methods Modifier and Type Method Description static FilegenerateSegmentMetadataFile(PinotFS fileSystem, URI tarFileURI)Generate a segment metadata only tar file, which contains only metadata.properties and creation.meta file.static URIgenerateSegmentMetadataURI(String segmentTarPath, String segmentName)static URIgenerateSegmentTarURI(URI dirURI, URI fileURI, String prefix, String suffix)static Map<String,String>getSegmentUriToTarPathMap(URI outputDirURI, PushJobSpec pushSpec, String[] files)static voidpushSegments(SegmentGenerationJobSpec spec, PinotFS fileSystem, List<String> tarFilePaths)static voidpushSegments(SegmentGenerationJobSpec spec, PinotFS fileSystem, List<String> tarFilePaths, List<org.apache.http.Header> headers, List<org.apache.http.NameValuePair> parameters)static voidsendSegmentUriAndMetadata(SegmentGenerationJobSpec spec, PinotFS fileSystem, Map<String,String> segmentUriToTarPathMap)This method takes a map of segment downloadURI to corresponding tar file path, and push those segments in metadata mode.static voidsendSegmentUriAndMetadata(SegmentGenerationJobSpec spec, PinotFS fileSystem, Map<String,String> segmentUriToTarPathMap, List<org.apache.http.Header> headers, List<org.apache.http.NameValuePair> parameters)This method takes a map of segment downloadURI to corresponding tar file path, and push those segments in metadata mode.static voidsendSegmentUris(SegmentGenerationJobSpec spec, List<String> segmentUris)static voidsendSegmentUris(SegmentGenerationJobSpec spec, List<String> segmentUris, List<org.apache.http.Header> headers, List<org.apache.http.NameValuePair> parameters)
-
-
-
Method Detail
-
generateSegmentTarURI
public static URI generateSegmentTarURI(URI dirURI, URI fileURI, String prefix, String suffix)
-
pushSegments
public static void pushSegments(SegmentGenerationJobSpec spec, PinotFS fileSystem, List<String> tarFilePaths) throws RetriableOperationException, AttemptsExceededException
-
sendSegmentUris
public static void sendSegmentUris(SegmentGenerationJobSpec spec, List<String> segmentUris) throws RetriableOperationException, AttemptsExceededException
-
sendSegmentUriAndMetadata
public static void sendSegmentUriAndMetadata(SegmentGenerationJobSpec spec, PinotFS fileSystem, Map<String,String> segmentUriToTarPathMap) throws Exception
This method takes a map of segment downloadURI to corresponding tar file path, and push those segments in metadata mode. The steps are: 1. Download segment from tar file path; 2. Untar segment metadata and creation meta files from the tar file to a segment metadata directory; 3. Tar this segment metadata directory into a tar file 4. Generate a POST request with segmentDownloadURI in header to push tar file to Pinot controller.- Parameters:
spec- is the segment generation job specfileSystem- is the PinotFs used to copy segment tar filesegmentUriToTarPathMap- contains the map of segment DownloadURI to segment tar file path- Throws:
Exception
-
pushSegments
public static void pushSegments(SegmentGenerationJobSpec spec, PinotFS fileSystem, List<String> tarFilePaths, List<org.apache.http.Header> headers, List<org.apache.http.NameValuePair> parameters) throws RetriableOperationException, AttemptsExceededException
-
sendSegmentUris
public static void sendSegmentUris(SegmentGenerationJobSpec spec, List<String> segmentUris, List<org.apache.http.Header> headers, List<org.apache.http.NameValuePair> parameters) throws RetriableOperationException, AttemptsExceededException
-
sendSegmentUriAndMetadata
public static void sendSegmentUriAndMetadata(SegmentGenerationJobSpec spec, PinotFS fileSystem, Map<String,String> segmentUriToTarPathMap, List<org.apache.http.Header> headers, List<org.apache.http.NameValuePair> parameters) throws Exception
This method takes a map of segment downloadURI to corresponding tar file path, and push those segments in metadata mode. The steps are: 1. Download segment from tar file path; 2. Untar segment metadata and creation meta files from the tar file to a segment metadata directory; 3. Tar this segment metadata directory into a tar file 4. Generate a POST request with segmentDownloadURI in header to push tar file to Pinot controller.- Parameters:
spec- is the segment generation job specfileSystem- is the PinotFs used to copy segment tar filesegmentUriToTarPathMap- contains the map of segment DownloadURI to segment tar file path- Throws:
Exception
-
getSegmentUriToTarPathMap
public static Map<String,String> getSegmentUriToTarPathMap(URI outputDirURI, PushJobSpec pushSpec, String[] files)
-
generateSegmentMetadataFile
public static File generateSegmentMetadataFile(PinotFS fileSystem, URI tarFileURI) throws Exception
Generate a segment metadata only tar file, which contains only metadata.properties and creation.meta file. The purpose of this is to create a lean tar to push to Pinot controller for adding segments without downloading the complete segment and untar the segment tarball. 1. Download segment tar file to temp dir; 2. Extract only metadata.properties and creation.meta files from the segment tar file; 3. Tar both files into a segment metadata file.- Throws:
Exception
-
generateSegmentMetadataURI
public static URI generateSegmentMetadataURI(String segmentTarPath, String segmentName) throws URISyntaxException
- Throws:
URISyntaxException
-
-