Package org.apache.druid.segment.realtime.appenderator

Class PeonAppenderatorsManager

java.lang.Object

org.apache.druid.segment.realtime.appenderator.PeonAppenderatorsManager

All Implemented Interfaces:

AppenderatorsManager

public class PeonAppenderatorsManager
extends Object
implements AppenderatorsManager

Manages Appenderators for tasks running within a CliPeon process. It provides the ability to create a realtime appenderator or multiple batch appenderators, and serves queries on the realtime appenderator. The implementation contains sanity checks that throw errors if more than one realtime appenderator is created, or if a task tries to create both realtime and batch appenderators. These checks can be adjusted if these assumptions are no longer true. Because the peon is a separate process that will terminate after task completion, this implementation relies on process shutdown for resource cleanup.

Constructor Summary

Constructors

Constructor	Description
PeonAppenderatorsManager()

Method Summary

Modifier and Type	Method	Description
Appenderator	createClosedSegmentsOfflineAppenderatorForTask(String taskId, DataSchema schema, AppenderatorConfig config, FireDepartmentMetrics metrics, org.apache.druid.segment.loading.DataSegmentPusher dataSegmentPusher, com.fasterxml.jackson.databind.ObjectMapper objectMapper, org.apache.druid.segment.IndexIO indexIO, org.apache.druid.segment.IndexMerger indexMerger, org.apache.druid.segment.incremental.RowIngestionMeters rowIngestionMeters, org.apache.druid.segment.incremental.ParseExceptionHandler parseExceptionHandler, boolean useMaxMemoryEstimates)
Appenderator	createOfflineAppenderatorForTask(String taskId, DataSchema schema, AppenderatorConfig config, FireDepartmentMetrics metrics, org.apache.druid.segment.loading.DataSegmentPusher dataSegmentPusher, com.fasterxml.jackson.databind.ObjectMapper objectMapper, org.apache.druid.segment.IndexIO indexIO, org.apache.druid.segment.IndexMerger indexMerger, org.apache.druid.segment.incremental.RowIngestionMeters rowIngestionMeters, org.apache.druid.segment.incremental.ParseExceptionHandler parseExceptionHandler, boolean useMaxMemoryEstimates)
Appenderator	createOpenSegmentsOfflineAppenderatorForTask(String taskId, DataSchema schema, AppenderatorConfig config, FireDepartmentMetrics metrics, org.apache.druid.segment.loading.DataSegmentPusher dataSegmentPusher, com.fasterxml.jackson.databind.ObjectMapper objectMapper, org.apache.druid.segment.IndexIO indexIO, org.apache.druid.segment.IndexMerger indexMerger, org.apache.druid.segment.incremental.RowIngestionMeters rowIngestionMeters, org.apache.druid.segment.incremental.ParseExceptionHandler parseExceptionHandler, boolean useMaxMemoryEstimates)	Creates an Appenderator suited for batch ingestion.
Appenderator	createRealtimeAppenderatorForTask(SegmentLoaderConfig segmentLoaderConfig, String taskId, DataSchema schema, AppenderatorConfig config, FireDepartmentMetrics metrics, org.apache.druid.segment.loading.DataSegmentPusher dataSegmentPusher, com.fasterxml.jackson.databind.ObjectMapper jsonMapper, org.apache.druid.segment.IndexIO indexIO, org.apache.druid.segment.IndexMerger indexMerger, org.apache.druid.query.QueryRunnerFactoryConglomerate conglomerate, DataSegmentAnnouncer segmentAnnouncer, org.apache.druid.java.util.emitter.service.ServiceEmitter emitter, org.apache.druid.query.QueryProcessingPool queryProcessingPool, org.apache.druid.segment.join.JoinableFactory joinableFactory, Cache cache, CacheConfig cacheConfig, CachePopulatorStats cachePopulatorStats, org.apache.druid.segment.incremental.RowIngestionMeters rowIngestionMeters, org.apache.druid.segment.incremental.ParseExceptionHandler parseExceptionHandler, boolean useMaxMemoryEstimates, CentralizedDatasourceSchemaConfig centralizedDatasourceSchemaConfig)	Creates an Appenderator suited for realtime ingestion.
<T> org.apache.druid.query.QueryRunner<T>	getQueryRunnerForIntervals(org.apache.druid.query.Query<T> query, Iterable<org.joda.time.Interval> intervals)	Returns a query runner for the given intervals over the Appenderators managed by this AppenderatorsManager.
<T> org.apache.druid.query.QueryRunner<T>	getQueryRunnerForSegments(org.apache.druid.query.Query<T> query, Iterable<org.apache.druid.query.SegmentDescriptor> specs)	Returns a query runner for the given segment specs over the Appenderators managed by this AppenderatorsManager.
void	removeAppenderatorsForTask(String taskId, String dataSource)	Removes any internal Appenderator-tracking state associated with the provided taskId.
boolean	shouldTaskMakeNodeAnnouncements()	As AppenderatorsManager implementions are service dependent (i.e., Peons and Indexers have different impls), this method allows Tasks to know whether they should announce themselves as nodes and segment servers to the rest of the cluster.
void	shutdown()	Shut down the AppenderatorsManager.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PeonAppenderatorsManager

public PeonAppenderatorsManager()

Method Detail

createRealtimeAppenderatorForTask

public Appenderator createRealtimeAppenderatorForTask(SegmentLoaderConfig segmentLoaderConfig,
                                                      String taskId,
                                                      DataSchema schema,
                                                      AppenderatorConfig config,
                                                      FireDepartmentMetrics metrics,
                                                      org.apache.druid.segment.loading.DataSegmentPusher dataSegmentPusher,
                                                      com.fasterxml.jackson.databind.ObjectMapper jsonMapper,
                                                      org.apache.druid.segment.IndexIO indexIO,
                                                      org.apache.druid.segment.IndexMerger indexMerger,
                                                      org.apache.druid.query.QueryRunnerFactoryConglomerate conglomerate,
                                                      DataSegmentAnnouncer segmentAnnouncer,
                                                      org.apache.druid.java.util.emitter.service.ServiceEmitter emitter,
                                                      org.apache.druid.query.QueryProcessingPool queryProcessingPool,
                                                      org.apache.druid.segment.join.JoinableFactory joinableFactory,
                                                      Cache cache,
                                                      CacheConfig cacheConfig,
                                                      CachePopulatorStats cachePopulatorStats,
                                                      org.apache.druid.segment.incremental.RowIngestionMeters rowIngestionMeters,
                                                      org.apache.druid.segment.incremental.ParseExceptionHandler parseExceptionHandler,
                                                      boolean useMaxMemoryEstimates,
                                                      CentralizedDatasourceSchemaConfig centralizedDatasourceSchemaConfig)

Description copied from interface: AppenderatorsManager

Creates an Appenderator suited for realtime ingestion. Note that this method's parameters include objects used for query processing.

Specified by:

createRealtimeAppenderatorForTask in interface AppenderatorsManager

createOfflineAppenderatorForTask

public Appenderator createOfflineAppenderatorForTask(String taskId,
                                                     DataSchema schema,
                                                     AppenderatorConfig config,
                                                     FireDepartmentMetrics metrics,
                                                     org.apache.druid.segment.loading.DataSegmentPusher dataSegmentPusher,
                                                     com.fasterxml.jackson.databind.ObjectMapper objectMapper,
                                                     org.apache.druid.segment.IndexIO indexIO,
                                                     org.apache.druid.segment.IndexMerger indexMerger,
                                                     org.apache.druid.segment.incremental.RowIngestionMeters rowIngestionMeters,
                                                     org.apache.druid.segment.incremental.ParseExceptionHandler parseExceptionHandler,
                                                     boolean useMaxMemoryEstimates)

Specified by:

createOfflineAppenderatorForTask in interface AppenderatorsManager

createOpenSegmentsOfflineAppenderatorForTask

public Appenderator createOpenSegmentsOfflineAppenderatorForTask(String taskId,
                                                                 DataSchema schema,
                                                                 AppenderatorConfig config,
                                                                 FireDepartmentMetrics metrics,
                                                                 org.apache.druid.segment.loading.DataSegmentPusher dataSegmentPusher,
                                                                 com.fasterxml.jackson.databind.ObjectMapper objectMapper,
                                                                 org.apache.druid.segment.IndexIO indexIO,
                                                                 org.apache.druid.segment.IndexMerger indexMerger,
                                                                 org.apache.druid.segment.incremental.RowIngestionMeters rowIngestionMeters,
                                                                 org.apache.druid.segment.incremental.ParseExceptionHandler parseExceptionHandler,
                                                                 boolean useMaxMemoryEstimates)

Description copied from interface: AppenderatorsManager

Creates an Appenderator suited for batch ingestion.

Specified by:

createOpenSegmentsOfflineAppenderatorForTask in interface AppenderatorsManager

createClosedSegmentsOfflineAppenderatorForTask

public Appenderator createClosedSegmentsOfflineAppenderatorForTask(String taskId,
                                                                   DataSchema schema,
                                                                   AppenderatorConfig config,
                                                                   FireDepartmentMetrics metrics,
                                                                   org.apache.druid.segment.loading.DataSegmentPusher dataSegmentPusher,
                                                                   com.fasterxml.jackson.databind.ObjectMapper objectMapper,
                                                                   org.apache.druid.segment.IndexIO indexIO,
                                                                   org.apache.druid.segment.IndexMerger indexMerger,
                                                                   org.apache.druid.segment.incremental.RowIngestionMeters rowIngestionMeters,
                                                                   org.apache.druid.segment.incremental.ParseExceptionHandler parseExceptionHandler,
                                                                   boolean useMaxMemoryEstimates)

Specified by:

createClosedSegmentsOfflineAppenderatorForTask in interface AppenderatorsManager

removeAppenderatorsForTask

public void removeAppenderatorsForTask(String taskId,
                                       String dataSource)

Description copied from interface: AppenderatorsManager

Removes any internal Appenderator-tracking state associated with the provided taskId. This method should be called when a task is finished using its Appenderators that were previously created by createRealtimeAppenderatorForTask or createOfflineAppenderatorForTask. The method can be called by the entity managing Tasks when the Tasks finish, such as ThreadingTaskRunner.

Specified by:

removeAppenderatorsForTask in interface AppenderatorsManager

getQueryRunnerForIntervals

public <T> org.apache.druid.query.QueryRunner<T> getQueryRunnerForIntervals(org.apache.druid.query.Query<T> query,
                                                                            Iterable<org.joda.time.Interval> intervals)

Description copied from interface: AppenderatorsManager

Returns a query runner for the given intervals over the Appenderators managed by this AppenderatorsManager.

Specified by:

getQueryRunnerForIntervals in interface AppenderatorsManager

getQueryRunnerForSegments

public <T> org.apache.druid.query.QueryRunner<T> getQueryRunnerForSegments(org.apache.druid.query.Query<T> query,
                                                                           Iterable<org.apache.druid.query.SegmentDescriptor> specs)

Description copied from interface: AppenderatorsManager

Returns a query runner for the given segment specs over the Appenderators managed by this AppenderatorsManager.

Specified by:

getQueryRunnerForSegments in interface AppenderatorsManager

shouldTaskMakeNodeAnnouncements

public boolean shouldTaskMakeNodeAnnouncements()

Description copied from interface: AppenderatorsManager

As AppenderatorsManager implementions are service dependent (i.e., Peons and Indexers have different impls), this method allows Tasks to know whether they should announce themselves as nodes and segment servers to the rest of the cluster. Only Tasks running in Peons (i.e., as separate processes) should make their own individual node announcements.

Specified by:

shouldTaskMakeNodeAnnouncements in interface AppenderatorsManager

shutdown

public void shutdown()

Description copied from interface: AppenderatorsManager

Shut down the AppenderatorsManager.

Specified by:

shutdown in interface AppenderatorsManager