Package org.apache.pinot.segment.local.upsert

Class BasePartitionUpsertMetadataManager

java.lang.Object

org.apache.pinot.segment.local.upsert.BasePartitionUpsertMetadataManager

All Implemented Interfaces:

Closeable, AutoCloseable, PartitionUpsertMetadataManager

Direct Known Subclasses:

ConcurrentMapPartitionUpsertMetadataManager

@ThreadSafe
public abstract class BasePartitionUpsertMetadataManager
extends Object
implements PartitionUpsertMetadataManager

Field Summary

Fields

Modifier and Type	Field	Description
protected List<String>	_comparisonColumns
protected String	_deleteRecordColumn
protected boolean	_enableSnapshot
protected boolean	_gotFirstConsumingSegment
protected HashFunction	_hashFunction
protected double	_largestSeenComparisonValue
protected long	_lastOutOfOrderEventReportTimeNs
protected org.slf4j.Logger	_logger
protected double	_metadataTTL
protected int	_numOutOfOrderEvents
protected AtomicInteger	_numPendingOperations
protected PartialUpsertHandler	_partialUpsertHandler
protected int	_partitionId
protected List<String>	_primaryKeyColumns
protected ServerMetrics	_serverMetrics
protected ReadWriteLock	_snapshotLock
protected boolean	_stopped
protected File	_tableIndexDir
protected String	_tableNameWithType
protected Set<IndexSegment>	_trackedSegments
protected static long	OUT_OF_ORDER_EVENT_MIN_REPORT_INTERVAL_NS

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	BasePartitionUpsertMetadataManager(String tableNameWithType, int partitionId, List<String> primaryKeyColumns, List<String> comparisonColumns, String deleteRecordColumn, HashFunction hashFunction, PartialUpsertHandler partialUpsertHandler, boolean enableSnapshot, double metadataTTL, File tableIndexDir, ServerMetrics serverMetrics)

Method Summary

Modifier and Type	Method	Description
protected abstract void	addOrReplaceSegment(ImmutableSegmentImpl segment, ThreadSafeMutableRoaringBitmap validDocIds, ThreadSafeMutableRoaringBitmap queryableDocIds, Iterator<RecordInfo> recordInfoIterator, IndexSegment oldSegment, org.roaringbitmap.buffer.MutableRoaringBitmap validDocIdsForOldSegment)
void	addRecord(MutableSegment segment, RecordInfo recordInfo)	Updates the upsert metadata for a new consumed record in the given consuming segment.
void	addSegment(ImmutableSegmentImpl segment, ThreadSafeMutableRoaringBitmap validDocIds, ThreadSafeMutableRoaringBitmap queryableDocIds, Iterator<RecordInfo> recordInfoIterator)	NOTE: We allow passing in validDocIds and queryableDocIds here so that the value can be easily accessed from the tests.
void	addSegment(ImmutableSegmentImpl segment, ThreadSafeMutableRoaringBitmap validDocIds, ThreadSafeMutableRoaringBitmap queryableDocIds, Iterator<RecordInfo> recordInfoIterator, boolean isPreloading)
void	addSegment(ImmutableSegment segment)	Initializes the upsert metadata for the given immutable segment.
protected void	addSegmentWithoutUpsert(ImmutableSegmentImpl segment, ThreadSafeMutableRoaringBitmap validDocIds, ThreadSafeMutableRoaringBitmap queryableDocIds, Iterator<RecordInfo> recordInfoIterator)
void	close()
protected void	deleteWatermark()	Deletes the watermark file.
protected abstract void	doAddRecord(MutableSegment segment, RecordInfo recordInfo)
protected void	doAddSegment(ImmutableSegmentImpl segment)
protected void	doClose()
protected abstract void	doRemoveExpiredPrimaryKeys()	Removes all primary keys that have comparison value smaller than (largestSeenComparisonValue - TTL).
protected void	doRemoveSegment(IndexSegment segment)
protected void	doReplaceSegment(ImmutableSegment segment, IndexSegment oldSegment)
protected void	doTakeSnapshot()
protected abstract GenericRow	doUpdateRecord(GenericRow record, RecordInfo recordInfo)
protected void	finishOperation()
protected abstract long	getNumPrimaryKeys()
List<String>	getPrimaryKeyColumns()	Returns the primary key columns.
protected File	getWatermarkFile()
protected void	handleOutOfOrderEvent(Object currentComparisonValue, Object recordComparisonValue)
protected double	loadWatermark()	Loads watermark from the file if exists.
protected void	persistWatermark(double watermark)	Persists watermark to the file.
void	preloadSegment(ImmutableSegment segment)	Different from adding a segment, when preloading a segment, the upsert metadata may be updated more efficiently.
void	removeExpiredPrimaryKeys()	Remove the expired primary keys from the metadata when TTL is enabled.
void	removeSegment(IndexSegment segment)	Removes the upsert metadata for the given segment.
protected abstract void	removeSegment(IndexSegment segment, org.roaringbitmap.buffer.MutableRoaringBitmap validDocIds)
void	replaceSegment(ImmutableSegment segment, ThreadSafeMutableRoaringBitmap validDocIds, ThreadSafeMutableRoaringBitmap queryableDocIds, Iterator<RecordInfo> recordInfoIterator, IndexSegment oldSegment)	NOTE: We allow passing in validDocIds and queryableDocIds here so that the value can be easily accessed from the tests.
void	replaceSegment(ImmutableSegment segment, IndexSegment oldSegment)	Replaces the upsert metadata for the old segment with the new immutable segment.
protected void	startOperation()
void	stop()	Stops the metadata manager.
void	takeSnapshot()	Takes snapshot for all the tracked immutable segments when snapshot is enabled.
GenericRow	updateRecord(GenericRow record, RecordInfo recordInfo)	Returns the merged record when partial-upsert is enabled.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

OUT_OF_ORDER_EVENT_MIN_REPORT_INTERVAL_NS

protected static final long OUT_OF_ORDER_EVENT_MIN_REPORT_INTERVAL_NS

_tableNameWithType

protected final String _tableNameWithType

_partitionId

protected final int _partitionId

_primaryKeyColumns

protected final List<String> _primaryKeyColumns

_comparisonColumns

protected final List<String> _comparisonColumns

_deleteRecordColumn

protected final String _deleteRecordColumn

_hashFunction

protected final HashFunction _hashFunction

_partialUpsertHandler

protected final PartialUpsertHandler _partialUpsertHandler

_enableSnapshot

protected final boolean _enableSnapshot

_metadataTTL

protected final double _metadataTTL

_tableIndexDir

protected final File _tableIndexDir

_serverMetrics

protected final ServerMetrics _serverMetrics

_logger

protected final org.slf4j.Logger _logger

_trackedSegments

protected final Set<IndexSegment> _trackedSegments

_gotFirstConsumingSegment

protected volatile boolean _gotFirstConsumingSegment

_snapshotLock

protected final ReadWriteLock _snapshotLock

_stopped

protected volatile boolean _stopped

_numPendingOperations

protected final AtomicInteger _numPendingOperations

_lastOutOfOrderEventReportTimeNs

protected long _lastOutOfOrderEventReportTimeNs

_numOutOfOrderEvents

protected int _numOutOfOrderEvents

_largestSeenComparisonValue

protected volatile double _largestSeenComparisonValue

Constructor Detail

BasePartitionUpsertMetadataManager

protected BasePartitionUpsertMetadataManager(String tableNameWithType,
                                             int partitionId,
                                             List<String> primaryKeyColumns,
                                             List<String> comparisonColumns,
                                             @Nullable
                                             String deleteRecordColumn,
                                             HashFunction hashFunction,
                                             @Nullable
                                             PartialUpsertHandler partialUpsertHandler,
                                             boolean enableSnapshot,
                                             double metadataTTL,
                                             File tableIndexDir,
                                             ServerMetrics serverMetrics)

Method Detail

getPrimaryKeyColumns

public List<String> getPrimaryKeyColumns()

Description copied from interface: PartitionUpsertMetadataManager

Returns the primary key columns.

Specified by:

getPrimaryKeyColumns in interface PartitionUpsertMetadataManager

addSegment

public void addSegment(ImmutableSegment segment)

Description copied from interface: PartitionUpsertMetadataManager

Initializes the upsert metadata for the given immutable segment.

Specified by:

addSegment in interface PartitionUpsertMetadataManager

doAddSegment

protected void doAddSegment(ImmutableSegmentImpl segment)

preloadSegment

public void preloadSegment(ImmutableSegment segment)

Description copied from interface: PartitionUpsertMetadataManager

Different from adding a segment, when preloading a segment, the upsert metadata may be updated more efficiently. Basically the upsert metadata can be directly updated for each primary key, without doing the more costly read-compare-update.

Specified by:

preloadSegment in interface PartitionUpsertMetadataManager

addSegment

public void addSegment(ImmutableSegmentImpl segment,
                       @Nullable
                       ThreadSafeMutableRoaringBitmap validDocIds,
                       @Nullable
                       ThreadSafeMutableRoaringBitmap queryableDocIds,
                       Iterator<RecordInfo> recordInfoIterator)

NOTE: We allow passing in validDocIds and queryableDocIds here so that the value can be easily accessed from the tests. The passed in bitmaps should always be empty.

addSegment

public void addSegment(ImmutableSegmentImpl segment,
                       @Nullable
                       ThreadSafeMutableRoaringBitmap validDocIds,
                       @Nullable
                       ThreadSafeMutableRoaringBitmap queryableDocIds,
                       Iterator<RecordInfo> recordInfoIterator,
                       boolean isPreloading)

getNumPrimaryKeys

protected abstract long getNumPrimaryKeys()

addOrReplaceSegment

protected abstract void addOrReplaceSegment(ImmutableSegmentImpl segment,
                                            ThreadSafeMutableRoaringBitmap validDocIds,
                                            @Nullable
                                            ThreadSafeMutableRoaringBitmap queryableDocIds,
                                            Iterator<RecordInfo> recordInfoIterator,
                                            @Nullable
                                            IndexSegment oldSegment,
                                            @Nullable
                                            org.roaringbitmap.buffer.MutableRoaringBitmap validDocIdsForOldSegment)

addSegmentWithoutUpsert

protected void addSegmentWithoutUpsert(ImmutableSegmentImpl segment,
                                       ThreadSafeMutableRoaringBitmap validDocIds,
                                       @Nullable
                                       ThreadSafeMutableRoaringBitmap queryableDocIds,
                                       Iterator<RecordInfo> recordInfoIterator)

addRecord

public void addRecord(MutableSegment segment,
                      RecordInfo recordInfo)

Description copied from interface: PartitionUpsertMetadataManager

Updates the upsert metadata for a new consumed record in the given consuming segment.

Specified by:

addRecord in interface PartitionUpsertMetadataManager

doAddRecord

protected abstract void doAddRecord(MutableSegment segment,
                                    RecordInfo recordInfo)

replaceSegment

public void replaceSegment(ImmutableSegment segment,
                           IndexSegment oldSegment)

Description copied from interface: PartitionUpsertMetadataManager

Replaces the upsert metadata for the old segment with the new immutable segment.

Specified by:

replaceSegment in interface PartitionUpsertMetadataManager

doReplaceSegment

protected void doReplaceSegment(ImmutableSegment segment,
                                IndexSegment oldSegment)

replaceSegment

public void replaceSegment(ImmutableSegment segment,
                           @Nullable
                           ThreadSafeMutableRoaringBitmap validDocIds,
                           @Nullable
                           ThreadSafeMutableRoaringBitmap queryableDocIds,
                           @Nullable
                           Iterator<RecordInfo> recordInfoIterator,
                           IndexSegment oldSegment)

NOTE: We allow passing in validDocIds and queryableDocIds here so that the value can be easily accessed from the tests. The passed in bitmaps should always be empty.

removeSegment

protected abstract void removeSegment(IndexSegment segment,
                                      org.roaringbitmap.buffer.MutableRoaringBitmap validDocIds)

removeSegment

public void removeSegment(IndexSegment segment)

Description copied from interface: PartitionUpsertMetadataManager

Removes the upsert metadata for the given segment.

Specified by:

removeSegment in interface PartitionUpsertMetadataManager

doRemoveSegment

protected void doRemoveSegment(IndexSegment segment)

updateRecord

public GenericRow updateRecord(GenericRow record,
                               RecordInfo recordInfo)

Description copied from interface: PartitionUpsertMetadataManager

Returns the merged record when partial-upsert is enabled.

Specified by:

updateRecord in interface PartitionUpsertMetadataManager

doUpdateRecord

protected abstract GenericRow doUpdateRecord(GenericRow record,
                                             RecordInfo recordInfo)

handleOutOfOrderEvent

protected void handleOutOfOrderEvent(Object currentComparisonValue,
                                     Object recordComparisonValue)

takeSnapshot

public void takeSnapshot()

Description copied from interface: PartitionUpsertMetadataManager

Takes snapshot for all the tracked immutable segments when snapshot is enabled. This method should be invoked before a new consuming segment starts consuming.

Specified by:

takeSnapshot in interface PartitionUpsertMetadataManager

doTakeSnapshot

protected void doTakeSnapshot()

loadWatermark

protected double loadWatermark()

Loads watermark from the file if exists.

persistWatermark

protected void persistWatermark(double watermark)

Persists watermark to the file.

deleteWatermark

protected void deleteWatermark()

Deletes the watermark file.

getWatermarkFile

protected File getWatermarkFile()

startOperation

protected void startOperation()

finishOperation

protected void finishOperation()

removeExpiredPrimaryKeys

public void removeExpiredPrimaryKeys()

Description copied from interface: PartitionUpsertMetadataManager

Remove the expired primary keys from the metadata when TTL is enabled.

Specified by:

removeExpiredPrimaryKeys in interface PartitionUpsertMetadataManager

doRemoveExpiredPrimaryKeys

protected abstract void doRemoveExpiredPrimaryKeys()

Removes all primary keys that have comparison value smaller than (largestSeenComparisonValue - TTL).

stop

public void stop()

Description copied from interface: PartitionUpsertMetadataManager

Stops the metadata manager. After invoking this method, no access to the metadata will be accepted.

Specified by:

stop in interface PartitionUpsertMetadataManager

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Throws:

IOException

doClose

protected void doClose()
                throws IOException

Throws:

IOException