Package org.apache.pinot.segment.local.data.manager

Interface TableDataManager

  • @ThreadSafe
public interface TableDataManager
    The TableDataManager interface provides APIs to manage segments under a table.

    • Method Detail

      • init

        void init​(TableDataManagerConfig tableDataManagerConfig,
          String instanceId,
          org.apache.helix.store.zk.ZkHelixPropertyStore<org.apache.helix.zookeeper.datamodel.ZNRecord> propertyStore,
          ServerMetrics serverMetrics,
          org.apache.helix.HelixManager helixManager,
          @Nullable
          ExecutorService segmentPreloadExecutor,
          @Nullable
          com.google.common.cache.LoadingCache<org.apache.commons.lang3.tuple.Pair<String,​String>,​SegmentErrorInfo> errorCache,
          TableDataManagerParams tableDataManagerParams)
        Initializes the table data manager. Should be called only once and before calling any other method.

      • start

        void start()
        Starts the table data manager. Should be called only once after table data manager gets initialized but before calling any other method.

      • shutDown

        void shutDown()
        Shuts down the table data manager. After calling shut down, no other method should be called. NOTE: Shut down might be called multiple times. The implementation should be able to handle that.

      • isShutDown

        boolean isShutDown()

      • addSegment

        void addSegment​(ImmutableSegment immutableSegment)
        Adds a loaded immutable segment into the table.

      • reloadSegment

        void reloadSegment​(String segmentName,
                   IndexLoadingConfig indexLoadingConfig,
                   SegmentZKMetadata zkMetadata,
                   SegmentMetadata localMetadata,
                   @Nullable
                   Schema schema,
                   boolean forceDownload)
            throws Exception
        Reloads an existing immutable segment for the table, which can be an OFFLINE or REALTIME table. A new segment may be downloaded if the local one has a different CRC; or can be forced to download if forceDownload flag is true. This operation is conducted within a failure handling framework and made transparent to ongoing queries, because the segment is in online serving state. TODO: Clean up this method to use the schema from the IndexLoadingConfig
        Parameters:
        segmentName - the segment to reload
        indexLoadingConfig - the latest table config to load segment
        zkMetadata - the segment metadata from zookeeper
        localMetadata - the segment metadata object held by server right now, which must not be null to reload the segment
        schema - the latest table schema to load segment
        forceDownload - whether to force to download raw segment to reload
        Throws:
        Exception - thrown upon failure when to reload the segment

      • addOrReplaceSegment

        void addOrReplaceSegment​(String segmentName,
                         IndexLoadingConfig indexLoadingConfig,
                         SegmentZKMetadata zkMetadata,
                         @Nullable
                         SegmentMetadata localMetadata)
                  throws Exception
        Adds or replaces an immutable segment for the table, which can be an OFFLINE or REALTIME table. A new segment may be downloaded if the local one has a different CRC or doesn't work as expected. This operation is conducted outside the failure handling framework as used in segment reloading, because the segment is not yet online serving queries, e.g. this method is used to add a new segment, or transition a segment to online serving state.
        Parameters:
        segmentName - the segment to add or replace
        indexLoadingConfig - the latest table config to load segment
        zkMetadata - the segment metadata from zookeeper
        localMetadata - the segment metadata object held by server, which can be null when the server is restarted or the segment is newly added to the table
        Throws:
        Exception - thrown upon failure when to add or replace the segment

      • removeSegment

        void removeSegment​(String segmentName)
        Removes a segment from the table.

      • tryLoadExistingSegment

        boolean tryLoadExistingSegment​(String segmentName,
                               IndexLoadingConfig indexLoadingConfig,
                               SegmentZKMetadata zkMetadata)
        Try to load a segment from an existing segment directory managed by the server. The segment loading may fail because the directory may not exist any more, or the segment data has a different crc now, or the existing segment data got corrupted etc.
        Returns:
        true if the segment is loaded successfully from the existing segment directory; false otherwise.

      • getSegmentDataDir

        File getSegmentDataDir​(String segmentName,
                       @Nullable
                       String segmentTier,
                       TableConfig tableConfig)
        Get the segment data directory, considering the segment tier if provided.

      • isSegmentDeletedRecently

        boolean isSegmentDeletedRecently​(String segmentName)
        Returns true if the segment was deleted in the last few minutes.

      • acquireSegments

        List<SegmentDataManager> acquireSegments​(List<String> segmentNames,
                                         List<String> missingSegments)
        Acquires the segments with the given segment names.

        It is the caller's responsibility to return the segments by calling releaseSegment(SegmentDataManager). This method may return some recently deleted segments in missingSegments. The caller can identify those segments by using isSegmentDeletedRecently(String).

        Parameters:
        segmentNames - List of names of the segment to acquire
        missingSegments - Holder for segments unable to be acquired
        Returns:
        List of segment data managers

      • acquireSegment

        @Nullable
SegmentDataManager acquireSegment​(String segmentName)
        Acquires the segments with the given segment name.

        It is the caller's responsibility to return the segments by calling releaseSegment(SegmentDataManager).

        Parameters:
        segmentName - Name of the segment to acquire
        Returns:
        Segment data manager with the given name, or null if no segment matches the name

      • releaseSegment

        void releaseSegment​(SegmentDataManager segmentDataManager)
        Releases the acquired segment.
        Parameters:
        segmentDataManager - Segment data manager

      • getNumSegments

        int getNumSegments()
        Returns the number of segments managed by this instance.

      • getTableName

        String getTableName()
        Returns the table name managed by this instance.

      • getTableDataDir

        File getTableDataDir()
        Returns the dir which contains the data segments.

      • getTableDataManagerConfig

        TableDataManagerConfig getTableDataManagerConfig()
        Returns the config for the table data manager.

      • addSegmentError

        void addSegmentError​(String segmentName,
                     SegmentErrorInfo segmentErrorInfo)
        Add error related to segment, if any. The implementation is expected to cache last 'N' errors for the table, related to segment transitions.

      • onConsumingToDropped

        default void onConsumingToDropped​(String segmentNameStr)
        Interface to handle segment state transitions from CONSUMING to DROPPED
        Parameters:
        segmentNameStr - name of segment for which the state change is being handled

      • onConsumingToOnline

        default void onConsumingToOnline​(String segmentNameStr)
        Interface to handle segment state transitions from CONSUMING to ONLINE
        Parameters:
        segmentNameStr - name of segment for which the state change is being handled