Package org.apache.pinot.segment.local.upsert

Interface PartitionUpsertMetadataManager

  • All Superinterfaces:
    AutoCloseable, Closeable
    All Known Implementing Classes:
    BasePartitionUpsertMetadataManager, ConcurrentMapPartitionUpsertMetadataManager
    @ThreadSafe
public interface PartitionUpsertMetadataManager
extends Closeable
    Manages the upsert metadata per partition.

    For multiple records with the same comparison value (default to timestamp), the manager will preserve the latest record based on the sequence number of the segment. If 2 records with the same comparison value are in the same segment, the one with larger doc id will be preserved. Note that for tables with sorted column, the records will be re-ordered when committing the segment, and we will use the re-ordered doc ids instead of the ingestion doc ids to decide the record to preserve.

    There will be short term inconsistency when updating the upsert metadata, but should be consistent after the operation is done:

    • When updating a new record, it first removes the doc id from the current location, then update the new location.
    • When adding a new segment, it removes the doc ids from the current locations before the segment being added to the RealtimeTableDataManager.
    • When replacing an existing segment, after the record location being replaced with the new segment, the following updates applied to the new segment's valid doc ids won't be reflected to the replaced segment's valid doc ids.

    • Method Detail

      • getPrimaryKeyColumns

        List<String> getPrimaryKeyColumns()
        Returns the primary key columns.

      • addSegment

        void addSegment​(ImmutableSegment segment)
        Initializes the upsert metadata for the given immutable segment.

      • preloadSegment

        void preloadSegment​(ImmutableSegment segment)
        Different from adding a segment, when preloading a segment, the upsert metadata may be updated more efficiently. Basically the upsert metadata can be directly updated for each primary key, without doing the more costly read-compare-update.

      • addRecord

        void addRecord​(MutableSegment segment,
               RecordInfo recordInfo)
        Updates the upsert metadata for a new consumed record in the given consuming segment.

      • replaceSegment

        void replaceSegment​(ImmutableSegment segment,
                    IndexSegment oldSegment)
        Replaces the upsert metadata for the old segment with the new immutable segment.

      • removeSegment

        void removeSegment​(IndexSegment segment)
        Removes the upsert metadata for the given segment.

      • takeSnapshot

        void takeSnapshot()
        Takes snapshot for all the tracked immutable segments when snapshot is enabled. This method should be invoked before a new consuming segment starts consuming.

      • removeExpiredPrimaryKeys

        void removeExpiredPrimaryKeys()
        Remove the expired primary keys from the metadata when TTL is enabled.

      • stop

        void stop()
        Stops the metadata manager. After invoking this method, no access to the metadata will be accepted.