Package org.apache.pinot.segment.local.upsert

Interface PartitionUpsertMetadataManager

  • All Superinterfaces:
    AutoCloseable, Closeable
    All Known Implementing Classes:
    BasePartitionUpsertMetadataManager, ConcurrentMapPartitionUpsertMetadataManager
    @ThreadSafe
public interface PartitionUpsertMetadataManager
extends Closeable
    Manages the upsert metadata per partition.

    For multiple records with the same comparison value (default to timestamp), the manager will preserve the latest record based on the sequence number of the segment. If 2 records with the same comparison value are in the same segment, the one with larger doc id will be preserved. Note that for tables with sorted column, the records will be re-ordered when committing the segment, and we will use the re-ordered doc ids instead of the ingestion doc ids to decide the record to preserve.

    There will be short term inconsistency when updating the upsert metadata, but should be consistent after the operation is done:

    • When updating a new record, it first removes the doc id from the current location, then update the new location.
    • When adding a new segment, it removes the doc ids from the current locations before the segment being added to the RealtimeTableDataManager.
    • When replacing an existing segment, after the record location being replaced with the new segment, the following updates applied to the new segment's valid doc ids won't be reflected to the replaced segment's valid doc ids.

    • Method Detail

      • getPrimaryKeyColumns

        List<String> getPrimaryKeyColumns()
        Returns the primary key columns.

      • addSegment

        void addSegment​(ImmutableSegment segment)
        Initializes the upsert metadata for the given immutable segment.

      • addRecord

        void addRecord​(MutableSegment segment,
               RecordInfo recordInfo)
        Updates the upsert metadata for a new consumed record in the given consuming segment.

      • replaceSegment

        void replaceSegment​(ImmutableSegment segment,
                    IndexSegment oldSegment)
        Replaces the upsert metadata for the old segment with the new immutable segment.

      • removeSegment

        void removeSegment​(IndexSegment segment)
        Removes the upsert metadata for the given segment.