org.apache.druid.segment.incremental

Class IncrementalIndex<AggregatorType>

java.lang.Object

org.apache.druid.segment.AbstractIndex

org.apache.druid.segment.incremental.IncrementalIndex<AggregatorType>

All Implemented Interfaces:

Closeable, AutoCloseable, Iterable<Row>

Direct Known Subclasses:

OffheapIncrementalIndex, OnheapIncrementalIndex

public abstract class IncrementalIndex<AggregatorType>
extends AbstractIndex
implements Iterable<Row>, Closeable

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	IncrementalIndex.Builder
static class	IncrementalIndex.DimensionDesc
static class	IncrementalIndex.MetricDesc

Field Summary

Fields

Modifier and Type	Field and Description
static Map<Object,ValueType>	TYPE_MAP

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	IncrementalIndex(IncrementalIndexSchema incrementalIndexSchema, boolean deserializeComplexMetrics, boolean reportParseExceptions, boolean concurrentEventAdd) Setting deserializeComplexMetrics to false is necessary for intermediate aggregation such as groupBy that should not deserialize input columns using ComplexMetricSerde for aggregators that return complex metrics.

Method Summary

Modifier and Type	Method and Description
IncrementalIndexAddResult	add(InputRow row) Adds a new row.
IncrementalIndexAddResult	add(InputRow row, boolean skipMaxRowsInMemoryCheck)
protected abstract org.apache.druid.segment.incremental.IncrementalIndex.AddToFactsResult	addToFacts(InputRow row, IncrementalIndexRow key, ThreadLocal<InputRow> rowContainer, com.google.common.base.Supplier<InputRow> rowSupplier, boolean skipMaxRowsInMemoryCheck)
abstract boolean	canAppendRow()
void	close()
protected Comparator<IncrementalIndexRow>	dimsComparator()
InputRow	formatRow(InputRow row)
AggregatorType[]	getAggs()
protected abstract AggregatorType[]	getAggsForRow(int rowOffset)
protected abstract Object	getAggVal(AggregatorType agg, int rowOffset, int aggPosition)
AtomicLong	getBytesInMemory()
ColumnCapabilities	getCapabilities(String column)
Map<String,ColumnCapabilitiesImpl>	getColumnCapabilities()
List<String>	getColumnNames()
static ParseException	getCombinedParseException(InputRow row, List<String> dimParseExceptionMessages, List<String> aggParseExceptionMessages)
IncrementalIndex.DimensionDesc	getDimension(String dimension)
Integer	getDimensionIndex(String dimension)
List<String>	getDimensionNames()
List<String>	getDimensionOrder()
List<IncrementalIndex.DimensionDesc>	getDimensions()
abstract org.apache.druid.segment.incremental.IncrementalIndex.FactsHolder	getFacts()
org.joda.time.Interval	getInterval()
abstract int	getLastRowIndex()
org.joda.time.DateTime	getMaxIngestedEventTime()
org.joda.time.DateTime	getMaxTime()
Metadata	getMetadata()
AggregatorFactory[]	getMetricAggs()
protected abstract double	getMetricDoubleValue(int rowOffset, int aggOffset)
protected abstract float	getMetricFloatValue(int rowOffset, int aggOffset)
protected abstract long	getMetricLongValue(int rowOffset, int aggOffset)
List<String>	getMetricNames()
protected abstract Object	getMetricObjectValue(int rowOffset, int aggOffset)
String	getMetricType(String metric)
org.joda.time.DateTime	getMinTime()
abstract String	getOutOfRowsReason()
protected abstract AggregatorType[]	initAggs(AggregatorFactory[] metrics, com.google.common.base.Supplier<InputRow> rowSupplier, boolean deserializeComplexMetrics, boolean concurrentEventAdd)
boolean	isEmpty()
protected abstract boolean	isNull(int rowOffset, int aggOffset)
boolean	isRollup()
Iterable<Row>	iterableWithPostAggregations(List<PostAggregator> postAggs, boolean descending)
Iterator<Row>	iterator()
void	loadDimensionIterable(Iterable<String> oldDimensionOrder, Map<String,ColumnCapabilitiesImpl> oldColumnCapabilities) Currently called to initialize IncrementalIndex dimension order during index creation Index dimension ordering could be changed to initialize from DimensionsSpec after resolution of https://github.com/apache/druid/issues/2011
protected ColumnSelectorFactory	makeColumnSelectorFactory(AggregatorFactory agg, com.google.common.base.Supplier<InputRow> in, boolean deserializeComplexMetrics)
static ColumnSelectorFactory	makeColumnSelectorFactory(VirtualColumns virtualColumns, AggregatorFactory agg, com.google.common.base.Supplier<InputRow> in, boolean deserializeComplexMetrics) Column selector used at ingestion time for inputs to aggregators.
ColumnValueSelector<?>	makeMetricColumnValueSelector(String metric, IncrementalIndexRowHolder currEntry)
int	size()
StorageAdapter	toStorageAdapter()

Methods inherited from class org.apache.druid.segment.AbstractIndex

toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Field Detail

TYPE_MAP

public static final Map<Object,ValueType> TYPE_MAP

Constructor Detail

IncrementalIndex

protected IncrementalIndex(IncrementalIndexSchema incrementalIndexSchema,
                           boolean deserializeComplexMetrics,
                           boolean reportParseExceptions,
                           boolean concurrentEventAdd)

Setting deserializeComplexMetrics to false is necessary for intermediate aggregation such as groupBy that should not deserialize input columns using ComplexMetricSerde for aggregators that return complex metrics.

Set concurrentEventAdd to true to indicate that adding of input row should be thread-safe (for example, groupBy where the multiple threads can add concurrently to the IncrementalIndex).

Parameters:

incrementalIndexSchema - the schema to use for incremental index

deserializeComplexMetrics - flag whether or not to call ComplexMetricExtractor.extractValue() on the input value for aggregators that return metrics other than float.

reportParseExceptions - flag whether or not to report ParseExceptions that occur while extracting values from input rows

concurrentEventAdd - flag whether ot not adding of input rows should be thread-safe

Method Detail

makeColumnSelectorFactory

public static ColumnSelectorFactory makeColumnSelectorFactory(VirtualColumns virtualColumns,
                                                              AggregatorFactory agg,
                                                              com.google.common.base.Supplier<InputRow> in,
                                                              boolean deserializeComplexMetrics)

Column selector used at ingestion time for inputs to aggregators.

Parameters:

agg - the aggregator

in - ingestion-time input row supplier

deserializeComplexMetrics - whether complex objects should be deserialized by a ComplexMetricExtractor

Returns:

column selector factory

isRollup

public boolean isRollup()

getFacts

public abstract org.apache.druid.segment.incremental.IncrementalIndex.FactsHolder getFacts()

canAppendRow

public abstract boolean canAppendRow()

getOutOfRowsReason

public abstract String getOutOfRowsReason()

initAggs

protected abstract AggregatorType[] initAggs(AggregatorFactory[] metrics,
                                             com.google.common.base.Supplier<InputRow> rowSupplier,
                                             boolean deserializeComplexMetrics,
                                             boolean concurrentEventAdd)

addToFacts

protected abstract org.apache.druid.segment.incremental.IncrementalIndex.AddToFactsResult addToFacts(InputRow row,
                                                                                                     IncrementalIndexRow key,
                                                                                                     ThreadLocal<InputRow> rowContainer,
                                                                                                     com.google.common.base.Supplier<InputRow> rowSupplier,
                                                                                                     boolean skipMaxRowsInMemoryCheck)
                                                                                              throws IndexSizeExceededException

Throws:

IndexSizeExceededException

getLastRowIndex

public abstract int getLastRowIndex()

getAggsForRow

protected abstract AggregatorType[] getAggsForRow(int rowOffset)

getAggVal

protected abstract Object getAggVal(AggregatorType agg,
                                    int rowOffset,
                                    int aggPosition)

getMetricFloatValue

protected abstract float getMetricFloatValue(int rowOffset,
                                             int aggOffset)

getMetricLongValue

protected abstract long getMetricLongValue(int rowOffset,
                                           int aggOffset)

getMetricObjectValue

protected abstract Object getMetricObjectValue(int rowOffset,
                                               int aggOffset)

getMetricDoubleValue

protected abstract double getMetricDoubleValue(int rowOffset,
                                               int aggOffset)

isNull

protected abstract boolean isNull(int rowOffset,
                                  int aggOffset)

close

public void close()

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

formatRow

public InputRow formatRow(InputRow row)

getColumnCapabilities

public Map<String,ColumnCapabilitiesImpl> getColumnCapabilities()

add

public IncrementalIndexAddResult add(InputRow row)
                              throws IndexSizeExceededException

Adds a new row. The row might correspond with another row that already exists, in which case this will update that row instead of inserting a new one.

Calls to add() are thread safe.

Parameters:

row - the row of data to add

Returns:

the number of rows in the data set after adding the InputRow

Throws:

IndexSizeExceededException

add

public IncrementalIndexAddResult add(InputRow row,
                                     boolean skipMaxRowsInMemoryCheck)
                              throws IndexSizeExceededException

Throws:

IndexSizeExceededException

getCombinedParseException

@Nullable
public static ParseException getCombinedParseException(InputRow row,
                                                                 @Nullable
                                                                 List<String> dimParseExceptionMessages,
                                                                 @Nullable
                                                                 List<String> aggParseExceptionMessages)

isEmpty

public boolean isEmpty()

size

public int size()

getBytesInMemory

public AtomicLong getBytesInMemory()

getAggs

public AggregatorType[] getAggs()

getMetricAggs

public AggregatorFactory[] getMetricAggs()

getDimensionNames

public List<String> getDimensionNames()

getDimensions

public List<IncrementalIndex.DimensionDesc> getDimensions()

getDimension

@Nullable
public IncrementalIndex.DimensionDesc getDimension(String dimension)

getMetricType

@Nullable
public String getMetricType(String metric)

makeMetricColumnValueSelector

public ColumnValueSelector<?> makeMetricColumnValueSelector(String metric,
                                                            IncrementalIndexRowHolder currEntry)

getInterval

public org.joda.time.Interval getInterval()

getMinTime

@Nullable
public org.joda.time.DateTime getMinTime()

getMaxTime

@Nullable
public org.joda.time.DateTime getMaxTime()

getDimensionIndex

@Nullable
public Integer getDimensionIndex(String dimension)

getDimensionOrder

public List<String> getDimensionOrder()

loadDimensionIterable

public void loadDimensionIterable(Iterable<String> oldDimensionOrder,
                                  Map<String,ColumnCapabilitiesImpl> oldColumnCapabilities)

Currently called to initialize IncrementalIndex dimension order during index creation Index dimension ordering could be changed to initialize from DimensionsSpec after resolution of https://github.com/apache/druid/issues/2011

getMetricNames

public List<String> getMetricNames()

getColumnNames

public List<String> getColumnNames()

Specified by:

getColumnNames in class AbstractIndex

toStorageAdapter

public StorageAdapter toStorageAdapter()

Specified by:

toStorageAdapter in class AbstractIndex

getCapabilities

@Nullable
public ColumnCapabilities getCapabilities(String column)

getMetadata

public Metadata getMetadata()

iterator

public Iterator<Row> iterator()

Specified by:

iterator in interface Iterable<Row>

iterableWithPostAggregations

public Iterable<Row> iterableWithPostAggregations(@Nullable
                                                  List<PostAggregator> postAggs,
                                                  boolean descending)

getMaxIngestedEventTime

public org.joda.time.DateTime getMaxIngestedEventTime()

makeColumnSelectorFactory

protected ColumnSelectorFactory makeColumnSelectorFactory(AggregatorFactory agg,
                                                          com.google.common.base.Supplier<InputRow> in,
                                                          boolean deserializeComplexMetrics)

dimsComparator

protected final Comparator<IncrementalIndexRow> dimsComparator()