Package org.apache.pinot.segment.local.upsert

Class BasePartitionUpsertMetadataManager

java.lang.Object

org.apache.pinot.segment.local.upsert.BasePartitionUpsertMetadataManager

All Implemented Interfaces:

Closeable, AutoCloseable, PartitionUpsertMetadataManager

Direct Known Subclasses:

ConcurrentMapPartitionUpsertMetadataManager

@ThreadSafe
public abstract class BasePartitionUpsertMetadataManager
extends Object
implements PartitionUpsertMetadataManager

Field Summary

Fields

Modifier and Type	Field	Description
protected List<String>	_comparisonColumns
protected UpsertContext	_context
protected double	_deletedKeysTTL
protected String	_deleteRecordColumn
protected boolean	_enableSnapshot
protected boolean	_gotFirstConsumingSegment
protected HashFunction	_hashFunction
protected com.google.common.util.concurrent.AtomicDouble	_largestSeenComparisonValue
protected long	_lastOutOfOrderEventReportTimeNs
protected org.slf4j.Logger	_logger
protected double	_metadataTTL
protected int	_numOutOfOrderEvents
protected PartialUpsertHandler	_partialUpsertHandler
protected int	_partitionId
protected List<String>	_primaryKeyColumns
protected ServerMetrics	_serverMetrics
protected ReadWriteLock	_snapshotLock
protected File	_tableIndexDir
protected String	_tableNameWithType
protected Set<IndexSegment>	_trackedSegments
protected static long	OUT_OF_ORDER_EVENT_MIN_REPORT_INTERVAL_NS

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	BasePartitionUpsertMetadataManager(String tableNameWithType, int partitionId, UpsertContext context)

Method Summary

Modifier and Type	Method	Description
protected abstract void	addOrReplaceSegment(ImmutableSegmentImpl segment, ThreadSafeMutableRoaringBitmap validDocIds, ThreadSafeMutableRoaringBitmap queryableDocIds, Iterator<RecordInfo> recordInfoIterator, IndexSegment oldSegment, org.roaringbitmap.buffer.MutableRoaringBitmap validDocIdsForOldSegment)
boolean	addRecord(MutableSegment segment, RecordInfo recordInfo)	Updates the upsert metadata for a new consumed record in the given consuming segment.
void	addSegment(ImmutableSegmentImpl segment, ThreadSafeMutableRoaringBitmap validDocIds, ThreadSafeMutableRoaringBitmap queryableDocIds, Iterator<RecordInfo> recordInfoIterator)	NOTE: We allow passing in validDocIds and queryableDocIds here so that the value can be easily accessed from the tests.
void	addSegment(ImmutableSegment segment)	Initializes the upsert metadata for the given immutable segment.
protected void	addSegmentWithoutUpsert(ImmutableSegmentImpl segment, ThreadSafeMutableRoaringBitmap validDocIds, ThreadSafeMutableRoaringBitmap queryableDocIds, Iterator<RecordInfo> recordInfoIterator)
void	close()
protected void	deleteWatermark()	Deletes the watermark file.
protected abstract boolean	doAddRecord(MutableSegment segment, RecordInfo recordInfo)	Returns true when the record is added to the upsert metadata manager, false when the record is out-of-order thus not added.
protected void	doAddSegment(ImmutableSegmentImpl segment)
protected void	doClose()
protected void	doPreloadSegment(ImmutableSegmentImpl segment)
protected abstract void	doRemoveExpiredPrimaryKeys()	Removes all primary keys that have comparison value smaller than (largestSeenComparisonValue - TTL).
protected void	doRemoveSegment(IndexSegment segment)
protected void	doReplaceSegment(ImmutableSegment segment, IndexSegment oldSegment)
protected void	doTakeSnapshot()
protected abstract GenericRow	doUpdateRecord(GenericRow record, RecordInfo recordInfo)
protected void	finishOperation()
protected abstract long	getNumPrimaryKeys()
List<String>	getPrimaryKeyColumns()	Returns the primary key columns.
protected org.roaringbitmap.buffer.MutableRoaringBitmap	getQueryableDocIds(IndexSegment segment, org.roaringbitmap.buffer.MutableRoaringBitmap validDocIds)
protected File	getWatermarkFile()
protected void	handleOutOfOrderEvent(Object currentComparisonValue, Object recordComparisonValue)
protected double	loadWatermark()	Loads watermark from the file if exists.
protected void	persistWatermark(double watermark)	Persists watermark to the file.
void	preloadSegment(ImmutableSegment segment)	Different from adding a segment, when preloading a segment, the upsert metadata may be updated more efficiently.
void	removeExpiredPrimaryKeys()	Remove the expired primary keys from the metadata when TTL is enabled.
void	removeSegment(IndexSegment segment)	Removes the upsert metadata for the given segment.
protected abstract void	removeSegment(IndexSegment segment, org.roaringbitmap.buffer.MutableRoaringBitmap validDocIds)
void	replaceSegment(ImmutableSegment segment, ThreadSafeMutableRoaringBitmap validDocIds, ThreadSafeMutableRoaringBitmap queryableDocIds, Iterator<RecordInfo> recordInfoIterator, IndexSegment oldSegment)	NOTE: We allow passing in validDocIds and queryableDocIds here so that the value can be easily accessed from the tests.
void	replaceSegment(ImmutableSegment segment, IndexSegment oldSegment)	Replaces the upsert metadata for the old segment with the new immutable segment.
protected boolean	startOperation()
void	stop()	Stops the metadata manager.
void	takeSnapshot()	Takes snapshot for all the tracked immutable segments when snapshot is enabled.
protected void	updatePrimaryKeyGauge()
protected void	updatePrimaryKeyGauge(long numPrimaryKeys)
GenericRow	updateRecord(GenericRow record, RecordInfo recordInfo)	Returns the merged record when partial-upsert is enabled.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

OUT_OF_ORDER_EVENT_MIN_REPORT_INTERVAL_NS

protected static final long OUT_OF_ORDER_EVENT_MIN_REPORT_INTERVAL_NS

_tableNameWithType

protected final String _tableNameWithType

_partitionId

protected final int _partitionId

_context

protected final UpsertContext _context

_primaryKeyColumns

protected final List<String> _primaryKeyColumns

_comparisonColumns

protected final List<String> _comparisonColumns

_deleteRecordColumn

protected final String _deleteRecordColumn

_hashFunction

protected final HashFunction _hashFunction

_partialUpsertHandler

protected final PartialUpsertHandler _partialUpsertHandler

_enableSnapshot

protected final boolean _enableSnapshot

_metadataTTL

protected final double _metadataTTL

_deletedKeysTTL

protected final double _deletedKeysTTL

_tableIndexDir

protected final File _tableIndexDir

_serverMetrics

protected final ServerMetrics _serverMetrics

_logger

protected final org.slf4j.Logger _logger

_trackedSegments

protected final Set<IndexSegment> _trackedSegments

_gotFirstConsumingSegment

protected volatile boolean _gotFirstConsumingSegment

_snapshotLock

protected final ReadWriteLock _snapshotLock

_lastOutOfOrderEventReportTimeNs

protected long _lastOutOfOrderEventReportTimeNs

_numOutOfOrderEvents

protected int _numOutOfOrderEvents

_largestSeenComparisonValue

protected final com.google.common.util.concurrent.AtomicDouble _largestSeenComparisonValue

Constructor Detail

BasePartitionUpsertMetadataManager

protected BasePartitionUpsertMetadataManager(String tableNameWithType,
                                             int partitionId,
                                             UpsertContext context)

Method Detail

getPrimaryKeyColumns

public List<String> getPrimaryKeyColumns()

Description copied from interface: PartitionUpsertMetadataManager

Returns the primary key columns.

Specified by:

getPrimaryKeyColumns in interface PartitionUpsertMetadataManager

getQueryableDocIds

@Nullable
protected org.roaringbitmap.buffer.MutableRoaringBitmap getQueryableDocIds(IndexSegment segment,
                                                                           org.roaringbitmap.buffer.MutableRoaringBitmap validDocIds)

addSegment

public void addSegment(ImmutableSegment segment)

Description copied from interface: PartitionUpsertMetadataManager

Initializes the upsert metadata for the given immutable segment.

Specified by:

addSegment in interface PartitionUpsertMetadataManager

doAddSegment

protected void doAddSegment(ImmutableSegmentImpl segment)

getNumPrimaryKeys

protected abstract long getNumPrimaryKeys()

updatePrimaryKeyGauge

protected void updatePrimaryKeyGauge(long numPrimaryKeys)

updatePrimaryKeyGauge

protected void updatePrimaryKeyGauge()

preloadSegment

public void preloadSegment(ImmutableSegment segment)

Description copied from interface: PartitionUpsertMetadataManager

Different from adding a segment, when preloading a segment, the upsert metadata may be updated more efficiently. Basically the upsert metadata can be directly updated for each primary key, without doing the more costly read-compare-update.

Specified by:

preloadSegment in interface PartitionUpsertMetadataManager

doPreloadSegment

protected void doPreloadSegment(ImmutableSegmentImpl segment)

addSegment

public void addSegment(ImmutableSegmentImpl segment,
                       @Nullable
                       ThreadSafeMutableRoaringBitmap validDocIds,
                       @Nullable
                       ThreadSafeMutableRoaringBitmap queryableDocIds,
                       Iterator<RecordInfo> recordInfoIterator)

NOTE: We allow passing in validDocIds and queryableDocIds here so that the value can be easily accessed from the tests. The passed in bitmaps should always be empty.

addOrReplaceSegment

protected abstract void addOrReplaceSegment(ImmutableSegmentImpl segment,
                                            ThreadSafeMutableRoaringBitmap validDocIds,
                                            @Nullable
                                            ThreadSafeMutableRoaringBitmap queryableDocIds,
                                            Iterator<RecordInfo> recordInfoIterator,
                                            @Nullable
                                            IndexSegment oldSegment,
                                            @Nullable
                                            org.roaringbitmap.buffer.MutableRoaringBitmap validDocIdsForOldSegment)

addSegmentWithoutUpsert

protected void addSegmentWithoutUpsert(ImmutableSegmentImpl segment,
                                       ThreadSafeMutableRoaringBitmap validDocIds,
                                       @Nullable
                                       ThreadSafeMutableRoaringBitmap queryableDocIds,
                                       Iterator<RecordInfo> recordInfoIterator)

addRecord

public boolean addRecord(MutableSegment segment,
                         RecordInfo recordInfo)

Description copied from interface: PartitionUpsertMetadataManager

Updates the upsert metadata for a new consumed record in the given consuming segment.

Specified by:

addRecord in interface PartitionUpsertMetadataManager

doAddRecord

protected abstract boolean doAddRecord(MutableSegment segment,
                                       RecordInfo recordInfo)

Returns true when the record is added to the upsert metadata manager, false when the record is out-of-order thus not added.

replaceSegment

public void replaceSegment(ImmutableSegment segment,
                           IndexSegment oldSegment)

Description copied from interface: PartitionUpsertMetadataManager

Replaces the upsert metadata for the old segment with the new immutable segment.

Specified by:

replaceSegment in interface PartitionUpsertMetadataManager

doReplaceSegment

protected void doReplaceSegment(ImmutableSegment segment,
                                IndexSegment oldSegment)

replaceSegment

public void replaceSegment(ImmutableSegment segment,
                           @Nullable
                           ThreadSafeMutableRoaringBitmap validDocIds,
                           @Nullable
                           ThreadSafeMutableRoaringBitmap queryableDocIds,
                           @Nullable
                           Iterator<RecordInfo> recordInfoIterator,
                           IndexSegment oldSegment)

NOTE: We allow passing in validDocIds and queryableDocIds here so that the value can be easily accessed from the tests. The passed in bitmaps should always be empty.

removeSegment

protected abstract void removeSegment(IndexSegment segment,
                                      org.roaringbitmap.buffer.MutableRoaringBitmap validDocIds)

removeSegment

public void removeSegment(IndexSegment segment)

Description copied from interface: PartitionUpsertMetadataManager

Removes the upsert metadata for the given segment.

Specified by:

removeSegment in interface PartitionUpsertMetadataManager

doRemoveSegment

protected void doRemoveSegment(IndexSegment segment)

updateRecord

public GenericRow updateRecord(GenericRow record,
                               RecordInfo recordInfo)

Description copied from interface: PartitionUpsertMetadataManager

Returns the merged record when partial-upsert is enabled.

Specified by:

updateRecord in interface PartitionUpsertMetadataManager

doUpdateRecord

protected abstract GenericRow doUpdateRecord(GenericRow record,
                                             RecordInfo recordInfo)

handleOutOfOrderEvent

protected void handleOutOfOrderEvent(Object currentComparisonValue,
                                     Object recordComparisonValue)

takeSnapshot

public void takeSnapshot()

Description copied from interface: PartitionUpsertMetadataManager

Takes snapshot for all the tracked immutable segments when snapshot is enabled. This method should be invoked before a new consuming segment starts consuming.

Specified by:

takeSnapshot in interface PartitionUpsertMetadataManager

doTakeSnapshot

protected void doTakeSnapshot()

loadWatermark

protected double loadWatermark()

Loads watermark from the file if exists.

persistWatermark

protected void persistWatermark(double watermark)

Persists watermark to the file.

deleteWatermark

protected void deleteWatermark()

Deletes the watermark file.

getWatermarkFile

protected File getWatermarkFile()

removeExpiredPrimaryKeys

public void removeExpiredPrimaryKeys()

Description copied from interface: PartitionUpsertMetadataManager

Remove the expired primary keys from the metadata when TTL is enabled.

Specified by:

removeExpiredPrimaryKeys in interface PartitionUpsertMetadataManager

doRemoveExpiredPrimaryKeys

protected abstract void doRemoveExpiredPrimaryKeys()

Removes all primary keys that have comparison value smaller than (largestSeenComparisonValue - TTL).

startOperation

protected boolean startOperation()

finishOperation

protected void finishOperation()

stop

public void stop()

Description copied from interface: PartitionUpsertMetadataManager

Stops the metadata manager. After invoking this method, no access to the metadata will be accepted.

Specified by:

stop in interface PartitionUpsertMetadataManager

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Throws:

IOException

doClose

protected void doClose()
                throws IOException

Throws:

IOException