Interface AppenderatorsManager

  • All Known Implementing Classes:
    DummyForInjectionAppenderatorsManager, PeonAppenderatorsManager, UnifiedIndexerAppenderatorsManager

    public interface AppenderatorsManager
    This interface defines entities that create and manage potentially multiple Appenderator instances. The AppenderatorsManager should be used by tasks running in a Peon or an CliIndexer process when it needs an Appenderator. The AppenderatorsManager also provides methods for creating QueryRunner instances that read the data held by the Appenderators created through the AppenderatorsManager. In later updates, this interface will be used to manage memory usage across multiple Appenderators, useful for the Indexer where all Tasks run in the same process. The methods on AppenderatorsManager can be called by multiple threads. This class provides similar functionality to the ServerManager and SegmentManager on the Historical processes.
    • Method Summary

      All Methods Instance Methods Abstract Methods 
      Modifier and Type Method Description
      Appenderator createClosedSegmentsOfflineAppenderatorForTask​(String taskId, DataSchema schema, AppenderatorConfig config, FireDepartmentMetrics metrics, org.apache.druid.segment.loading.DataSegmentPusher dataSegmentPusher, com.fasterxml.jackson.databind.ObjectMapper objectMapper, org.apache.druid.segment.IndexIO indexIO, org.apache.druid.segment.IndexMerger indexMerger, org.apache.druid.segment.incremental.RowIngestionMeters rowIngestionMeters, org.apache.druid.segment.incremental.ParseExceptionHandler parseExceptionHandler, boolean useMaxMemoryEstimates)  
      Appenderator createOfflineAppenderatorForTask​(String taskId, DataSchema schema, AppenderatorConfig config, FireDepartmentMetrics metrics, org.apache.druid.segment.loading.DataSegmentPusher dataSegmentPusher, com.fasterxml.jackson.databind.ObjectMapper objectMapper, org.apache.druid.segment.IndexIO indexIO, org.apache.druid.segment.IndexMerger indexMerger, org.apache.druid.segment.incremental.RowIngestionMeters rowIngestionMeters, org.apache.druid.segment.incremental.ParseExceptionHandler parseExceptionHandler, boolean useMaxMemoryEstimates)  
      Appenderator createOpenSegmentsOfflineAppenderatorForTask​(String taskId, DataSchema schema, AppenderatorConfig config, FireDepartmentMetrics metrics, org.apache.druid.segment.loading.DataSegmentPusher dataSegmentPusher, com.fasterxml.jackson.databind.ObjectMapper objectMapper, org.apache.druid.segment.IndexIO indexIO, org.apache.druid.segment.IndexMerger indexMerger, org.apache.druid.segment.incremental.RowIngestionMeters rowIngestionMeters, org.apache.druid.segment.incremental.ParseExceptionHandler parseExceptionHandler, boolean useMaxMemoryEstimates)
      Creates an Appenderator suited for batch ingestion.
      Appenderator createRealtimeAppenderatorForTask​(SegmentLoaderConfig segmentLoaderConfig, String taskId, DataSchema schema, AppenderatorConfig config, FireDepartmentMetrics metrics, org.apache.druid.segment.loading.DataSegmentPusher dataSegmentPusher, com.fasterxml.jackson.databind.ObjectMapper objectMapper, org.apache.druid.segment.IndexIO indexIO, org.apache.druid.segment.IndexMerger indexMerger, org.apache.druid.query.QueryRunnerFactoryConglomerate conglomerate, DataSegmentAnnouncer segmentAnnouncer, org.apache.druid.java.util.emitter.service.ServiceEmitter emitter, org.apache.druid.query.QueryProcessingPool queryProcessingPool, org.apache.druid.segment.join.JoinableFactory joinableFactory, Cache cache, CacheConfig cacheConfig, CachePopulatorStats cachePopulatorStats, org.apache.druid.segment.incremental.RowIngestionMeters rowIngestionMeters, org.apache.druid.segment.incremental.ParseExceptionHandler parseExceptionHandler, boolean useMaxMemoryEstimates, CentralizedDatasourceSchemaConfig centralizedDatasourceSchemaConfig)
      Creates an Appenderator suited for realtime ingestion.
      <T> org.apache.druid.query.QueryRunner<T> getQueryRunnerForIntervals​(org.apache.druid.query.Query<T> query, Iterable<org.joda.time.Interval> intervals)
      Returns a query runner for the given intervals over the Appenderators managed by this AppenderatorsManager.
      <T> org.apache.druid.query.QueryRunner<T> getQueryRunnerForSegments​(org.apache.druid.query.Query<T> query, Iterable<org.apache.druid.query.SegmentDescriptor> specs)
      Returns a query runner for the given segment specs over the Appenderators managed by this AppenderatorsManager.
      void removeAppenderatorsForTask​(String taskId, String dataSource)
      Removes any internal Appenderator-tracking state associated with the provided taskId.
      boolean shouldTaskMakeNodeAnnouncements()
      As AppenderatorsManager implementions are service dependent (i.e., Peons and Indexers have different impls), this method allows Tasks to know whether they should announce themselves as nodes and segment servers to the rest of the cluster.
      void shutdown()
      Shut down the AppenderatorsManager.
    • Method Detail

      • createRealtimeAppenderatorForTask

        Appenderator createRealtimeAppenderatorForTask​(SegmentLoaderConfig segmentLoaderConfig,
                                                       String taskId,
                                                       DataSchema schema,
                                                       AppenderatorConfig config,
                                                       FireDepartmentMetrics metrics,
                                                       org.apache.druid.segment.loading.DataSegmentPusher dataSegmentPusher,
                                                       com.fasterxml.jackson.databind.ObjectMapper objectMapper,
                                                       org.apache.druid.segment.IndexIO indexIO,
                                                       org.apache.druid.segment.IndexMerger indexMerger,
                                                       org.apache.druid.query.QueryRunnerFactoryConglomerate conglomerate,
                                                       DataSegmentAnnouncer segmentAnnouncer,
                                                       org.apache.druid.java.util.emitter.service.ServiceEmitter emitter,
                                                       org.apache.druid.query.QueryProcessingPool queryProcessingPool,
                                                       org.apache.druid.segment.join.JoinableFactory joinableFactory,
                                                       Cache cache,
                                                       CacheConfig cacheConfig,
                                                       CachePopulatorStats cachePopulatorStats,
                                                       org.apache.druid.segment.incremental.RowIngestionMeters rowIngestionMeters,
                                                       org.apache.druid.segment.incremental.ParseExceptionHandler parseExceptionHandler,
                                                       boolean useMaxMemoryEstimates,
                                                       CentralizedDatasourceSchemaConfig centralizedDatasourceSchemaConfig)
        Creates an Appenderator suited for realtime ingestion. Note that this method's parameters include objects used for query processing.
      • createOpenSegmentsOfflineAppenderatorForTask

        Appenderator createOpenSegmentsOfflineAppenderatorForTask​(String taskId,
                                                                  DataSchema schema,
                                                                  AppenderatorConfig config,
                                                                  FireDepartmentMetrics metrics,
                                                                  org.apache.druid.segment.loading.DataSegmentPusher dataSegmentPusher,
                                                                  com.fasterxml.jackson.databind.ObjectMapper objectMapper,
                                                                  org.apache.druid.segment.IndexIO indexIO,
                                                                  org.apache.druid.segment.IndexMerger indexMerger,
                                                                  org.apache.druid.segment.incremental.RowIngestionMeters rowIngestionMeters,
                                                                  org.apache.druid.segment.incremental.ParseExceptionHandler parseExceptionHandler,
                                                                  boolean useMaxMemoryEstimates)
        Creates an Appenderator suited for batch ingestion.
      • createClosedSegmentsOfflineAppenderatorForTask

        Appenderator createClosedSegmentsOfflineAppenderatorForTask​(String taskId,
                                                                    DataSchema schema,
                                                                    AppenderatorConfig config,
                                                                    FireDepartmentMetrics metrics,
                                                                    org.apache.druid.segment.loading.DataSegmentPusher dataSegmentPusher,
                                                                    com.fasterxml.jackson.databind.ObjectMapper objectMapper,
                                                                    org.apache.druid.segment.IndexIO indexIO,
                                                                    org.apache.druid.segment.IndexMerger indexMerger,
                                                                    org.apache.druid.segment.incremental.RowIngestionMeters rowIngestionMeters,
                                                                    org.apache.druid.segment.incremental.ParseExceptionHandler parseExceptionHandler,
                                                                    boolean useMaxMemoryEstimates)
      • createOfflineAppenderatorForTask

        Appenderator createOfflineAppenderatorForTask​(String taskId,
                                                      DataSchema schema,
                                                      AppenderatorConfig config,
                                                      FireDepartmentMetrics metrics,
                                                      org.apache.druid.segment.loading.DataSegmentPusher dataSegmentPusher,
                                                      com.fasterxml.jackson.databind.ObjectMapper objectMapper,
                                                      org.apache.druid.segment.IndexIO indexIO,
                                                      org.apache.druid.segment.IndexMerger indexMerger,
                                                      org.apache.druid.segment.incremental.RowIngestionMeters rowIngestionMeters,
                                                      org.apache.druid.segment.incremental.ParseExceptionHandler parseExceptionHandler,
                                                      boolean useMaxMemoryEstimates)
      • removeAppenderatorsForTask

        void removeAppenderatorsForTask​(String taskId,
                                        String dataSource)
        Removes any internal Appenderator-tracking state associated with the provided taskId. This method should be called when a task is finished using its Appenderators that were previously created by createRealtimeAppenderatorForTask or createOfflineAppenderatorForTask. The method can be called by the entity managing Tasks when the Tasks finish, such as ThreadingTaskRunner.
      • getQueryRunnerForIntervals

        <T> org.apache.druid.query.QueryRunner<T> getQueryRunnerForIntervals​(org.apache.druid.query.Query<T> query,
                                                                             Iterable<org.joda.time.Interval> intervals)
        Returns a query runner for the given intervals over the Appenderators managed by this AppenderatorsManager.
      • getQueryRunnerForSegments

        <T> org.apache.druid.query.QueryRunner<T> getQueryRunnerForSegments​(org.apache.druid.query.Query<T> query,
                                                                            Iterable<org.apache.druid.query.SegmentDescriptor> specs)
        Returns a query runner for the given segment specs over the Appenderators managed by this AppenderatorsManager.
      • shouldTaskMakeNodeAnnouncements

        boolean shouldTaskMakeNodeAnnouncements()
        As AppenderatorsManager implementions are service dependent (i.e., Peons and Indexers have different impls), this method allows Tasks to know whether they should announce themselves as nodes and segment servers to the rest of the cluster. Only Tasks running in Peons (i.e., as separate processes) should make their own individual node announcements.
      • shutdown

        void shutdown()
        Shut down the AppenderatorsManager.