Package org.apache.pinot.segment.local.upsert

Class BasePartitionUpsertMetadataManager

java.lang.Object

org.apache.pinot.segment.local.upsert.BasePartitionUpsertMetadataManager

All Implemented Interfaces:

Closeable, AutoCloseable, PartitionUpsertMetadataManager

Direct Known Subclasses:

ConcurrentMapPartitionUpsertMetadataManager

@ThreadSafe
public abstract class BasePartitionUpsertMetadataManager
extends Object
implements PartitionUpsertMetadataManager

Field Summary

Fields

Modifier and Type	Field	Description
protected boolean	_closed
protected String	_comparisonColumn
protected boolean	_enableSnapshot
protected HashFunction	_hashFunction
protected long	_lastOutOfOrderEventReportTimeNs
protected org.slf4j.Logger	_logger
protected int	_numOutOfOrderEvents
protected PartialUpsertHandler	_partialUpsertHandler
protected int	_partitionId
protected List<String>	_primaryKeyColumns
Set<IndexSegment>	_replacedSegments
protected ServerMetrics	_serverMetrics
protected String	_tableNameWithType
protected static long	OUT_OF_ORDER_EVENT_MIN_REPORT_INTERVAL_NS

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	BasePartitionUpsertMetadataManager(String tableNameWithType, int partitionId, List<String> primaryKeyColumns, String comparisonColumn, HashFunction hashFunction, PartialUpsertHandler partialUpsertHandler, boolean enableSnapshot, ServerMetrics serverMetrics)

Method Summary

Modifier and Type	Method	Description
protected abstract void	addOrReplaceSegment(ImmutableSegmentImpl segment, ThreadSafeMutableRoaringBitmap validDocIds, Iterator<RecordInfo> recordInfoIterator, IndexSegment oldSegment, org.roaringbitmap.buffer.MutableRoaringBitmap validDocIdsForOldSegment)
void	addSegment(ImmutableSegmentImpl segment, ThreadSafeMutableRoaringBitmap validDocIds, Iterator<RecordInfo> recordInfoIterator)	NOTE: We allow passing in validDocIds here so that the value can be easily accessed from the tests.
void	addSegment(ImmutableSegment segment)	Initializes the upsert metadata for the given immutable segment.
void	close()
protected abstract long	getNumPrimaryKeys()
List<String>	getPrimaryKeyColumns()	Returns the primary key columns.
protected void	handleOutOfOrderEvent(Object currentComparisonValue, Object recordComparisonValue)
void	removeSegment(IndexSegment segment)	Removes the upsert metadata for the given segment.
protected abstract void	removeSegment(IndexSegment segment, org.roaringbitmap.buffer.MutableRoaringBitmap validDocIds)
void	replaceSegment(ImmutableSegment segment, ThreadSafeMutableRoaringBitmap validDocIds, Iterator<RecordInfo> recordInfoIterator, IndexSegment oldSegment)	NOTE: We allow passing in validDocIds here so that the value can be easily accessed from the tests.
void	replaceSegment(ImmutableSegment segment, IndexSegment oldSegment)	Replaces the upsert metadata for the old segment with the new immutable segment.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.pinot.segment.local.upsert.PartitionUpsertMetadataManager

addRecord, updateRecord

Field Detail

OUT_OF_ORDER_EVENT_MIN_REPORT_INTERVAL_NS

protected static final long OUT_OF_ORDER_EVENT_MIN_REPORT_INTERVAL_NS

_tableNameWithType

protected final String _tableNameWithType

_partitionId

protected final int _partitionId

_primaryKeyColumns

protected final List<String> _primaryKeyColumns

_comparisonColumn

protected final String _comparisonColumn

_hashFunction

protected final HashFunction _hashFunction

_partialUpsertHandler

protected final PartialUpsertHandler _partialUpsertHandler

_enableSnapshot

protected final boolean _enableSnapshot

_serverMetrics

protected final ServerMetrics _serverMetrics

_logger

protected final org.slf4j.Logger _logger

_replacedSegments

public final Set<IndexSegment> _replacedSegments

_closed

protected volatile boolean _closed

_lastOutOfOrderEventReportTimeNs

protected long _lastOutOfOrderEventReportTimeNs

_numOutOfOrderEvents

protected int _numOutOfOrderEvents

Constructor Detail

BasePartitionUpsertMetadataManager

protected BasePartitionUpsertMetadataManager(String tableNameWithType,
                                             int partitionId,
                                             List<String> primaryKeyColumns,
                                             String comparisonColumn,
                                             HashFunction hashFunction,
                                             @Nullable
                                             PartialUpsertHandler partialUpsertHandler,
                                             boolean enableSnapshot,
                                             ServerMetrics serverMetrics)

Method Detail

getPrimaryKeyColumns

public List<String> getPrimaryKeyColumns()

Description copied from interface: PartitionUpsertMetadataManager

Returns the primary key columns.

Specified by:

getPrimaryKeyColumns in interface PartitionUpsertMetadataManager

addSegment

public void addSegment(ImmutableSegment segment)

Description copied from interface: PartitionUpsertMetadataManager

Initializes the upsert metadata for the given immutable segment.

Specified by:

addSegment in interface PartitionUpsertMetadataManager

addSegment

public void addSegment(ImmutableSegmentImpl segment,
                       @Nullable
                       ThreadSafeMutableRoaringBitmap validDocIds,
                       Iterator<RecordInfo> recordInfoIterator)

NOTE: We allow passing in validDocIds here so that the value can be easily accessed from the tests. The passed in validDocIds should always be empty.

getNumPrimaryKeys

protected abstract long getNumPrimaryKeys()

addOrReplaceSegment

protected abstract void addOrReplaceSegment(ImmutableSegmentImpl segment,
                                            ThreadSafeMutableRoaringBitmap validDocIds,
                                            Iterator<RecordInfo> recordInfoIterator,
                                            @Nullable
                                            IndexSegment oldSegment,
                                            @Nullable
                                            org.roaringbitmap.buffer.MutableRoaringBitmap validDocIdsForOldSegment)

replaceSegment

public void replaceSegment(ImmutableSegment segment,
                           IndexSegment oldSegment)

Description copied from interface: PartitionUpsertMetadataManager

Replaces the upsert metadata for the old segment with the new immutable segment.

Specified by:

replaceSegment in interface PartitionUpsertMetadataManager

replaceSegment

public void replaceSegment(ImmutableSegment segment,
                           @Nullable
                           ThreadSafeMutableRoaringBitmap validDocIds,
                           @Nullable
                           Iterator<RecordInfo> recordInfoIterator,
                           IndexSegment oldSegment)

NOTE: We allow passing in validDocIds here so that the value can be easily accessed from the tests. The passed in validDocIds should always be empty.

removeSegment

protected abstract void removeSegment(IndexSegment segment,
                                      org.roaringbitmap.buffer.MutableRoaringBitmap validDocIds)

removeSegment

public void removeSegment(IndexSegment segment)

Description copied from interface: PartitionUpsertMetadataManager

Removes the upsert metadata for the given segment.

Specified by:

removeSegment in interface PartitionUpsertMetadataManager

handleOutOfOrderEvent

protected void handleOutOfOrderEvent(Object currentComparisonValue,
                                     Object recordComparisonValue)

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Throws:

IOException