Package org.apache.druid.segment.incremental

Class IncrementalIndex

java.lang.Object

org.apache.druid.segment.incremental.IncrementalIndex

All Implemented Interfaces:

Closeable, AutoCloseable, Iterable<Row>, Expr.InputBindingInspector, ColumnInspector

Direct Known Subclasses:

OnheapIncrementalIndex

public abstract class IncrementalIndex
extends Object
implements Iterable<Row>, Closeable, ColumnInspector

In-memory, row-based data structure used to hold data during ingestion. Realtime tasks query this index using IncrementalIndexStorageAdapter. Concurrency model: add(InputRow) and add(InputRow, boolean) are not thread-safe, and must be called from a single thread or externally synchronized. However, the methods that support IncrementalIndexStorageAdapter are thread-safe, and may be called concurrently with each other, and with the "add" methods. This concurrency model supports real-time queries of the data in the index.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	IncrementalIndex.DimensionDesc
static interface	IncrementalIndex.FactsHolder
static class	IncrementalIndex.InputRowHolder
static class	IncrementalIndex.MetricDesc

Field Summary

Fields

Modifier and Type	Field	Description
protected boolean	preserveExistingMetrics

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	IncrementalIndex(IncrementalIndexSchema incrementalIndexSchema, boolean preserveExistingMetrics, boolean useMaxMemoryEstimates)

Method Summary

Modifier and Type	Method	Description
IncrementalIndexAddResult	add(InputRow row)	Adds a new row.
IncrementalIndexAddResult	add(InputRow row, boolean skipMaxRowsInMemoryCheck)	Adds a new row.
protected abstract org.apache.druid.segment.incremental.IncrementalIndex.AddToFactsResult	addToFacts(IncrementalIndexRow key, IncrementalIndex.InputRowHolder inputRowHolder, boolean skipMaxRowsInMemoryCheck)
abstract boolean	canAppendRow()
void	close()
protected Comparator<IncrementalIndexRow>	dimsComparator()
InputRow	formatRow(InputRow row)
AtomicLong	getBytesInMemory()
ColumnCapabilities	getColumnCapabilities(String columnName)	Returns capabilities of a particular column.
ColumnFormat	getColumnFormat(String columnName)
Map<String,ColumnFormat>	getColumnFormats()
List<String>	getColumnNames()
static ParseException	getCombinedParseException(InputRow row, List<String> dimParseExceptionMessages, List<String> aggParseExceptionMessages)
IncrementalIndex.DimensionDesc	getDimension(String dimension)
Integer	getDimensionIndex(String dimension)
List<String>	getDimensionNames()
List<String>	getDimensionOrder()
List<IncrementalIndex.DimensionDesc>	getDimensions()
DimensionsSpec	getDimensionsSpec()	Returns dimensionsSpec from the ingestionSpec.
abstract IncrementalIndex.FactsHolder	getFacts()
org.joda.time.Interval	getInterval()
abstract int	getLastRowIndex()
org.joda.time.DateTime	getMaxIngestedEventTime()
org.joda.time.DateTime	getMaxTime()
Metadata	getMetadata()
AggregatorFactory[]	getMetricAggs()
protected abstract double	getMetricDoubleValue(int rowOffset, int aggOffset)
protected abstract float	getMetricFloatValue(int rowOffset, int aggOffset)
protected abstract long	getMetricLongValue(int rowOffset, int aggOffset)
List<String>	getMetricNames()
protected abstract Object	getMetricObjectValue(int rowOffset, int aggOffset)
org.joda.time.DateTime	getMinTime()
abstract String	getOutOfRowsReason()
protected abstract void	initAggs(AggregatorFactory[] metrics, IncrementalIndex.InputRowHolder rowSupplier)
boolean	isEmpty()
protected abstract boolean	isNull(int rowOffset, int aggOffset)
boolean	isRollup()
abstract Iterable<Row>	iterableWithPostAggregations(List<PostAggregator> postAggs, boolean descending)
Iterator<Row>	iterator()
void	loadDimensionIterable(Iterable<String> oldDimensionOrder, Map<String,ColumnFormat> oldColumnCapabilities)	Currently called to initialize IncrementalIndex dimension order during index creation Index dimension ordering could be changed to initialize from DimensionsSpec after resolution of https://github.com/apache/druid/issues/2011
protected ColumnSelectorFactory	makeColumnSelectorFactory(AggregatorFactory agg, IncrementalIndex.InputRowHolder in)
static ColumnSelectorFactory	makeColumnSelectorFactory(VirtualColumns virtualColumns, IncrementalIndex.InputRowHolder inputRowHolder, AggregatorFactory agg)	Column selector used at ingestion time for inputs to aggregators.
static ColumnCapabilitiesImpl	makeDefaultCapabilitiesFromValueType(ColumnType type)
ColumnValueSelector<?>	makeMetricColumnValueSelector(String metric, IncrementalIndexRowHolder currEntry)
int	size()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.druid.segment.ColumnInspector

getType

Methods inherited from interface org.apache.druid.math.expr.Expr.InputBindingInspector

areNumeric, areNumeric, areSameTypes, areSameTypes, areScalar, areScalar, canVectorize, canVectorize

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Field Detail

preserveExistingMetrics

protected final boolean preserveExistingMetrics

Constructor Detail

IncrementalIndex

protected IncrementalIndex(IncrementalIndexSchema incrementalIndexSchema,
                           boolean preserveExistingMetrics,
                           boolean useMaxMemoryEstimates)

Parameters:

incrementalIndexSchema - the schema to use for incremental index

preserveExistingMetrics - When set to true, for any row that already has metric (with the same name defined in metricSpec), the metric aggregator in metricSpec is skipped and the existing metric is unchanged. If the row does not already have the metric, then the metric aggregator is applied on the source column as usual. This should only be set for DruidInputSource since that is the only case where we can have existing metrics. This is currently only use by auto compaction and should not be use for anything else.

useMaxMemoryEstimates - true if max values should be used to estimate memory

Method Detail

makeColumnSelectorFactory

public static ColumnSelectorFactory makeColumnSelectorFactory(VirtualColumns virtualColumns,
                                                              IncrementalIndex.InputRowHolder inputRowHolder,
                                                              @Nullable
                                                              AggregatorFactory agg)

Column selector used at ingestion time for inputs to aggregators.

Parameters:

virtualColumns - virtual columns

inputRowHolder - ingestion-time input row holder

agg - the aggregator, or null to make a generic aggregator. Only required if the agg has AggregatorFactory.getIntermediateType() as ValueType.COMPLEX, because in this case we need to do some magic to ensure the correct values show up.

Returns:

column selector factory

getFacts

public abstract IncrementalIndex.FactsHolder getFacts()

canAppendRow

public abstract boolean canAppendRow()

getOutOfRowsReason

public abstract String getOutOfRowsReason()

initAggs

protected abstract void initAggs(AggregatorFactory[] metrics,
                                 IncrementalIndex.InputRowHolder rowSupplier)

addToFacts

protected abstract org.apache.druid.segment.incremental.IncrementalIndex.AddToFactsResult addToFacts(IncrementalIndexRow key,
                                                                                                     IncrementalIndex.InputRowHolder inputRowHolder,
                                                                                                     boolean skipMaxRowsInMemoryCheck)
                                                                                              throws IndexSizeExceededException

Throws:

IndexSizeExceededException

getLastRowIndex

public abstract int getLastRowIndex()

getMetricFloatValue

protected abstract float getMetricFloatValue(int rowOffset,
                                             int aggOffset)

getMetricLongValue

protected abstract long getMetricLongValue(int rowOffset,
                                           int aggOffset)

getMetricObjectValue

protected abstract Object getMetricObjectValue(int rowOffset,
                                               int aggOffset)

getMetricDoubleValue

protected abstract double getMetricDoubleValue(int rowOffset,
                                               int aggOffset)

isNull

protected abstract boolean isNull(int rowOffset,
                                  int aggOffset)

isRollup

public boolean isRollup()

close

public void close()

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

formatRow

public InputRow formatRow(InputRow row)

getColumnFormats

public Map<String,ColumnFormat> getColumnFormats()

getColumnCapabilities

@Nullable
public ColumnCapabilities getColumnCapabilities(String columnName)

Description copied from interface: ColumnInspector

Returns capabilities of a particular column.

Specified by:

getColumnCapabilities in interface ColumnInspector

Parameters:

columnName - column name

Returns:

capabilities, or null

getColumnFormat

@Nullable
public ColumnFormat getColumnFormat(String columnName)

add

public IncrementalIndexAddResult add(InputRow row)
                              throws IndexSizeExceededException

Adds a new row. The row might correspond with another row that already exists, in which case this will update that row instead of inserting a new one. Not thread-safe.

Parameters:

row - the row of data to add

Returns:

the number of rows in the data set after adding the InputRow. If any parse failure occurs, a ParseException is returned in IncrementalIndexAddResult.

Throws:

IndexSizeExceededException - this exception is thrown once it reaches max rows limit and skipMaxRowsInMemoryCheck is set to false.

add

public IncrementalIndexAddResult add(InputRow row,
                                     boolean skipMaxRowsInMemoryCheck)
                              throws IndexSizeExceededException

Adds a new row. The row might correspond with another row that already exists, in which case this will update that row instead of inserting a new one. Not thread-safe.

Parameters:

row - the row of data to add

skipMaxRowsInMemoryCheck - whether or not to skip the check of rows exceeding the max rows or bytes limit

Returns:

the number of rows in the data set after adding the InputRow. If any parse failure occurs, a ParseException is returned in IncrementalIndexAddResult.

Throws:

IndexSizeExceededException - this exception is thrown once it reaches max rows limit and skipMaxRowsInMemoryCheck is set to false.

getCombinedParseException

@Nullable
public static ParseException getCombinedParseException(InputRow row,
                                                       @Nullable
                                                       List<String> dimParseExceptionMessages,
                                                       @Nullable
                                                       List<String> aggParseExceptionMessages)

isEmpty

public boolean isEmpty()

size

public int size()

getBytesInMemory

public AtomicLong getBytesInMemory()

getMetricAggs

public AggregatorFactory[] getMetricAggs()

getDimensionsSpec

public DimensionsSpec getDimensionsSpec()

Returns dimensionsSpec from the ingestionSpec.

getDimensionNames

public List<String> getDimensionNames()

getDimensions

public List<IncrementalIndex.DimensionDesc> getDimensions()

getDimension

@Nullable
public IncrementalIndex.DimensionDesc getDimension(String dimension)

makeMetricColumnValueSelector

public ColumnValueSelector<?> makeMetricColumnValueSelector(String metric,
                                                            IncrementalIndexRowHolder currEntry)

getInterval

public org.joda.time.Interval getInterval()

getMinTime

@Nullable
public org.joda.time.DateTime getMinTime()

getMaxTime

@Nullable
public org.joda.time.DateTime getMaxTime()

getDimensionIndex

@Nullable
public Integer getDimensionIndex(String dimension)

getDimensionOrder

public List<String> getDimensionOrder()

makeDefaultCapabilitiesFromValueType

public static ColumnCapabilitiesImpl makeDefaultCapabilitiesFromValueType(ColumnType type)

loadDimensionIterable

public void loadDimensionIterable(Iterable<String> oldDimensionOrder,
                                  Map<String,ColumnFormat> oldColumnCapabilities)

Currently called to initialize IncrementalIndex dimension order during index creation Index dimension ordering could be changed to initialize from DimensionsSpec after resolution of https://github.com/apache/druid/issues/2011

getMetricNames

public List<String> getMetricNames()

getColumnNames

public List<String> getColumnNames()

getMetadata

public Metadata getMetadata()

iterator

public Iterator<Row> iterator()

Specified by:

iterator in interface Iterable<Row>

iterableWithPostAggregations

public abstract Iterable<Row> iterableWithPostAggregations(@Nullable
                                                           List<PostAggregator> postAggs,
                                                           boolean descending)

getMaxIngestedEventTime

public org.joda.time.DateTime getMaxIngestedEventTime()

makeColumnSelectorFactory

protected ColumnSelectorFactory makeColumnSelectorFactory(@Nullable
                                                          AggregatorFactory agg,
                                                          IncrementalIndex.InputRowHolder in)

dimsComparator

protected final Comparator<IncrementalIndexRow> dimsComparator()