Package org.apache.druid.segment.realtime.plumber

Class RealtimePlumber

    • Constructor Detail

      • RealtimePlumber

        public RealtimePlumber​(DataSchema schema,
                       RealtimeTuningConfig config,
                       FireDepartmentMetrics metrics,
                       org.apache.druid.java.util.emitter.service.ServiceEmitter emitter,
                       org.apache.druid.query.QueryRunnerFactoryConglomerate conglomerate,
                       DataSegmentAnnouncer segmentAnnouncer,
                       org.apache.druid.query.QueryProcessingPool queryProcessingPool,
                       org.apache.druid.segment.join.JoinableFactory joinableFactory,
                       org.apache.druid.segment.loading.DataSegmentPusher dataSegmentPusher,
                       SegmentPublisher segmentPublisher,
                       SegmentHandoffNotifier handoffNotifier,
                       org.apache.druid.segment.IndexMerger indexMerger,
                       org.apache.druid.segment.IndexIO indexIO,
                       Cache cache,
                       CacheConfig cacheConfig,
                       CachePopulatorStats cachePopulatorStats,
                       com.fasterxml.jackson.databind.ObjectMapper objectMapper)

    • Method Detail

      • startJob

        public Object startJob()
        Description copied from interface: Plumber
        Perform any initial setup. Should be called before using any other methods, and should be paired with a corresponding call to Plumber.finishJob().
        Specified by:
        startJob in interface Plumber
        Returns:
        the metadata of the "newest" segment that might have previously been persisted

      • add

        public org.apache.druid.segment.incremental.IncrementalIndexAddResult add​(org.apache.druid.data.input.InputRow row,
                                                                          com.google.common.base.Supplier<org.apache.druid.data.input.Committer> committerSupplier)
                                                                   throws org.apache.druid.segment.incremental.IndexSizeExceededException
        Specified by:
        add in interface Plumber
        Parameters:
        row - the row to insert
        committerSupplier - supplier of a committer associated with all data that has been added, including this row
        Returns:
        IncrementalIndexAddResult whose rowCount - positive numbers indicate how many summarized rows exist in the index for that timestamp, -1 means a row was thrown away because it was too late -2 means a row was thrown away because it is duplicate
        Throws:
        org.apache.druid.segment.incremental.IndexSizeExceededException

      • getQueryRunner

        public <T> org.apache.druid.query.QueryRunner<T> getQueryRunner​(org.apache.druid.query.Query<T> query)
        Specified by:
        getQueryRunner in interface Plumber

      • persist

        public void persist​(org.apache.druid.data.input.Committer committer)
        Description copied from interface: Plumber
        Persist any in-memory indexed data to durable storage. This may be only somewhat durable, e.g. the machine's local disk.
        Specified by:
        persist in interface Plumber
        Parameters:
        committer - committer to use after persisting data

      • finishJob

        public void finishJob()
        Description copied from interface: Plumber
        Perform any final processing and clean up after ourselves. Should be called after all data has been fed into sinks and persisted.
        Specified by:
        finishJob in interface Plumber

      • initializeExecutors

        protected void initializeExecutors()

      • shutdownExecutors

        protected void shutdownExecutors()

      • bootstrapSinksFromDisk

        protected Object bootstrapSinksFromDisk()

      • startPersistThread

        protected void startPersistThread()

      • abandonSegment

        protected void abandonSegment​(long truncatedTime,
                              Sink sink)
        Unannounces a given sink and removes all local references to it. It is important that this is only called from the single-threaded mergeExecutor, since otherwise chaos may ensue if merged segments are deleted while being created.
        Parameters:
        truncatedTime - sink key
        sink - sink to unannounce

      • computeBaseDir

        protected File computeBaseDir​(DataSchema schema)

      • computeCorruptedFileDumpDir

        protected File computeCorruptedFileDumpDir​(File persistDir,
                                           DataSchema schema)

      • computePersistDir

        protected File computePersistDir​(DataSchema schema,
                                 org.joda.time.Interval interval)

      • persistHydrant

        protected int persistHydrant​(FireHydrant indexToPersist,
                             DataSchema schema,
                             org.joda.time.Interval interval,
                             Map<String,​Object> metadataElems)
        Persists the given hydrant and returns the number of rows persisted
        Parameters:
        indexToPersist - hydrant to persist
        schema - datasource schema
        interval - interval to persist
        Returns:
        the number of rows persisted