Package org.apache.pinot.segment.local.utils

Class IngestionUtils

java.lang.Object

org.apache.pinot.segment.local.utils.IngestionUtils

public final class IngestionUtils
extends Object

Helper methods for ingestion

Method Summary

Modifier and Type	Method	Description
static String	buildSegment(SegmentGeneratorConfig segmentGeneratorConfig)	Builds a segment using given SegmentGeneratorConfig
static Long	extractTimeValue(Comparable time)
static SegmentGeneratorConfig	generateSegmentGeneratorConfig(TableConfig tableConfig, Schema schema)	Create SegmentGeneratorConfig using tableConfig and schema.
static SegmentGeneratorConfig	generateSegmentGeneratorConfig(TableConfig tableConfig, Schema schema, BatchIngestionConfig batchIngestionConfig)	Create SegmentGeneratorConfig using tableConfig, schema and batchIngestionConfig.
static Set<String>	getFieldsForRecordExtractor(IngestionConfig ingestionConfig, Schema schema)	Extracts all fields required by the RecordExtractor from the given TableConfig and Schema Fields for ingestion come from 2 places: 1.
static PinotFS	getOutputPinotFS(BatchConfig batchConfig, URI fileURI)	Creates an instance of the PinotFS using the fileURI and fs properties from BatchConfig
static boolean	shouldIngestRow(GenericRow genericRow)	Returns false if the record contains key GenericRow.SKIP_RECORD_KEY with value true
static void	uploadSegment(String tableNameWithType, BatchConfig batchConfig, List<URI> segmentTarURIs, AuthProvider authProvider)	Uploads the segments from the provided segmentTar URIs to the table, using push details from the batchConfig

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

generateSegmentGeneratorConfig

public static SegmentGeneratorConfig generateSegmentGeneratorConfig(TableConfig tableConfig,
                                                                    Schema schema)
                                                             throws IOException,
                                                                    ClassNotFoundException

Create SegmentGeneratorConfig using tableConfig and schema. All properties are taken from the 1st Map in tableConfig -> ingestionConfig -> batchIngestionConfig -> batchConfigMaps

Parameters:

tableConfig - tableConfig with the batchConfigMap set

schema - pinot schema

Throws:

IOException

ClassNotFoundException

generateSegmentGeneratorConfig

public static SegmentGeneratorConfig generateSegmentGeneratorConfig(TableConfig tableConfig,
                                                                    Schema schema,
                                                                    BatchIngestionConfig batchIngestionConfig)
                                                             throws ClassNotFoundException,
                                                                    IOException

Create SegmentGeneratorConfig using tableConfig, schema and batchIngestionConfig. The provided batchIngestionConfig will take precedence over the one in tableConfig

Throws:

ClassNotFoundException

IOException

buildSegment

public static String buildSegment(SegmentGeneratorConfig segmentGeneratorConfig)
                           throws Exception

Builds a segment using given SegmentGeneratorConfig

Returns:

segment name

Throws:

Exception

uploadSegment

public static void uploadSegment(String tableNameWithType,
                                 BatchConfig batchConfig,
                                 List<URI> segmentTarURIs,
                                 @Nullable
                                 AuthProvider authProvider)
                          throws Exception

Uploads the segments from the provided segmentTar URIs to the table, using push details from the batchConfig

Parameters:

tableNameWithType - name of the table to upload the segment

batchConfig - batchConfig with details about push such as controllerURI, pushAttempts, pushParallelism, etc

segmentTarURIs - list of URI for the segment tar files

authProvider - auth provider

Throws:

Exception

getOutputPinotFS

public static PinotFS getOutputPinotFS(BatchConfig batchConfig,
                                       URI fileURI)

Creates an instance of the PinotFS using the fileURI and fs properties from BatchConfig

getFieldsForRecordExtractor

public static Set<String> getFieldsForRecordExtractor(@Nullable
                                                      IngestionConfig ingestionConfig,
                                                      Schema schema)

Extracts all fields required by the RecordExtractor from the given TableConfig and Schema Fields for ingestion come from 2 places: 1. The schema 2. The ingestion config in the table config. The ingestion config (e.g. filter, complexType) can have fields which are not in the schema.

shouldIngestRow

public static boolean shouldIngestRow(GenericRow genericRow)

Returns false if the record contains key GenericRow.SKIP_RECORD_KEY with value true

extractTimeValue

public static Long extractTimeValue(Comparable time)