Package org.apache.pinot.segment.local.upsert

Class ConcurrentMapPartitionUpsertMetadataManager

java.lang.Object

org.apache.pinot.segment.local.upsert.BasePartitionUpsertMetadataManager

org.apache.pinot.segment.local.upsert.ConcurrentMapPartitionUpsertMetadataManager

All Implemented Interfaces:

Closeable, AutoCloseable, PartitionUpsertMetadataManager

@ThreadSafe
public class ConcurrentMapPartitionUpsertMetadataManager
extends BasePartitionUpsertMetadataManager

Implementation of PartitionUpsertMetadataManager that is backed by a ConcurrentHashMap.

Field Summary

Fields inherited from class org.apache.pinot.segment.local.upsert.BasePartitionUpsertMetadataManager

_comparisonColumns, _deleteRecordColumn, _enableSnapshot, _gotFirstConsumingSegment, _hashFunction, _largestSeenComparisonValue, _lastOutOfOrderEventReportTimeNs, _logger, _metadataTTL, _numOutOfOrderEvents, _numPendingOperations, _partialUpsertHandler, _partitionId, _primaryKeyColumns, _serverMetrics, _snapshotLock, _stopped, _tableIndexDir, _tableNameWithType, _trackedSegments, OUT_OF_ORDER_EVENT_MIN_REPORT_INTERVAL_NS

Constructor Summary

Constructors

Constructor	Description
ConcurrentMapPartitionUpsertMetadataManager(String tableNameWithType, int partitionId, List<String> primaryKeyColumns, List<String> comparisonColumns, String deleteRecordColumn, HashFunction hashFunction, PartialUpsertHandler partialUpsertHandler, boolean enableSnapshot, double metadataTTL, File tableIndexDir, ServerMetrics serverMetrics)

Method Summary

Modifier and Type	Method	Description
protected void	addOrReplaceSegment(ImmutableSegmentImpl segment, ThreadSafeMutableRoaringBitmap validDocIds, ThreadSafeMutableRoaringBitmap queryableDocIds, Iterator<RecordInfo> recordInfoIterator, IndexSegment oldSegment, org.roaringbitmap.buffer.MutableRoaringBitmap validDocIdsForOldSegment)
protected void	addSegmentWithoutUpsert(ImmutableSegmentImpl segment, ThreadSafeMutableRoaringBitmap validDocIds, ThreadSafeMutableRoaringBitmap queryableDocIds, Iterator<RecordInfo> recordInfoIterator)
protected void	doAddRecord(MutableSegment segment, RecordInfo recordInfo)
void	doRemoveExpiredPrimaryKeys()	Removes all primary keys that have comparison value smaller than (largestSeenComparisonValue - TTL).
protected GenericRow	doUpdateRecord(GenericRow record, RecordInfo recordInfo)
protected long	getNumPrimaryKeys()
protected void	removeSegment(IndexSegment segment, org.roaringbitmap.buffer.MutableRoaringBitmap validDocIds)

Methods inherited from class org.apache.pinot.segment.local.upsert.BasePartitionUpsertMetadataManager

addRecord, addSegment, addSegment, addSegment, close, deleteWatermark, doAddSegment, doClose, doRemoveSegment, doReplaceSegment, doTakeSnapshot, finishOperation, getPrimaryKeyColumns, getWatermarkFile, handleOutOfOrderEvent, loadWatermark, persistWatermark, preloadSegment, removeExpiredPrimaryKeys, removeSegment, replaceSegment, replaceSegment, startOperation, stop, takeSnapshot, updateRecord

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ConcurrentMapPartitionUpsertMetadataManager

public ConcurrentMapPartitionUpsertMetadataManager(String tableNameWithType,
                                                   int partitionId,
                                                   List<String> primaryKeyColumns,
                                                   List<String> comparisonColumns,
                                                   @Nullable
                                                   String deleteRecordColumn,
                                                   HashFunction hashFunction,
                                                   @Nullable
                                                   PartialUpsertHandler partialUpsertHandler,
                                                   boolean enableSnapshot,
                                                   double metadataTTL,
                                                   File tableIndexDir,
                                                   ServerMetrics serverMetrics)

Method Detail

getNumPrimaryKeys

protected long getNumPrimaryKeys()

Specified by:

getNumPrimaryKeys in class BasePartitionUpsertMetadataManager

addOrReplaceSegment

protected void addOrReplaceSegment(ImmutableSegmentImpl segment,
                                   ThreadSafeMutableRoaringBitmap validDocIds,
                                   @Nullable
                                   ThreadSafeMutableRoaringBitmap queryableDocIds,
                                   Iterator<RecordInfo> recordInfoIterator,
                                   @Nullable
                                   IndexSegment oldSegment,
                                   @Nullable
                                   org.roaringbitmap.buffer.MutableRoaringBitmap validDocIdsForOldSegment)

Specified by:

addOrReplaceSegment in class BasePartitionUpsertMetadataManager

addSegmentWithoutUpsert

protected void addSegmentWithoutUpsert(ImmutableSegmentImpl segment,
                                       ThreadSafeMutableRoaringBitmap validDocIds,
                                       @Nullable
                                       ThreadSafeMutableRoaringBitmap queryableDocIds,
                                       Iterator<RecordInfo> recordInfoIterator)

Overrides:

addSegmentWithoutUpsert in class BasePartitionUpsertMetadataManager

removeSegment

protected void removeSegment(IndexSegment segment,
                             org.roaringbitmap.buffer.MutableRoaringBitmap validDocIds)

Specified by:

removeSegment in class BasePartitionUpsertMetadataManager

doRemoveExpiredPrimaryKeys

public void doRemoveExpiredPrimaryKeys()

Description copied from class: BasePartitionUpsertMetadataManager

Removes all primary keys that have comparison value smaller than (largestSeenComparisonValue - TTL).

Specified by:

doRemoveExpiredPrimaryKeys in class BasePartitionUpsertMetadataManager

doAddRecord

protected void doAddRecord(MutableSegment segment,
                           RecordInfo recordInfo)

Specified by:

doAddRecord in class BasePartitionUpsertMetadataManager

doUpdateRecord

protected GenericRow doUpdateRecord(GenericRow record,
                                    RecordInfo recordInfo)

Specified by:

doUpdateRecord in class BasePartitionUpsertMetadataManager