Package org.apache.pinot.segment.local.segment.index.dictionary

Class DictionaryIndexType

java.lang.Object

org.apache.pinot.segment.spi.index.AbstractIndexType<DictionaryIndexConfig,Dictionary,SegmentDictionaryCreator>

org.apache.pinot.segment.local.segment.index.dictionary.DictionaryIndexType

All Implemented Interfaces:

ConfigurableFromIndexLoadingConfig<DictionaryIndexConfig>, IndexType<DictionaryIndexConfig,Dictionary,SegmentDictionaryCreator>

public class DictionaryIndexType
extends AbstractIndexType<DictionaryIndexConfig,Dictionary,SegmentDictionaryCreator>
implements ConfigurableFromIndexLoadingConfig<DictionaryIndexConfig>

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.pinot.segment.spi.index.IndexType

IndexType.BuildLifecycle

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	DictionaryIndexType()

Method Summary

Modifier and Type	Method	Description
ColumnConfigDeserializer<DictionaryIndexConfig>	createDeserializer()
SegmentDictionaryCreator	createIndexCreator(IndexCreationContext context, DictionaryIndexConfig indexConfig)
SegmentDictionaryCreator	createIndexCreator(FieldSpec fieldSpec, File indexDir, boolean useVarLengthDictionary)
IndexHandler	createIndexHandler(SegmentDirectory segmentDirectory, Map<String,FieldIndexConfigs> configsByCol, Schema schema, TableConfig tableConfig)
static MutableDictionary	createMutableDictionary(MutableIndexContext context, DictionaryIndexConfig config)	Creates a MutableDictionary.
protected IndexReaderFactory<Dictionary>	createReaderFactory()
Map<String,DictionaryIndexConfig>	fromIndexLoadingConfig(IndexLoadingConfig indexLoadingConfig)	Returns a map that can be used to get the index config.
DictionaryIndexConfig	getDefaultConfig()
static String	getFileExtension()
List<String>	getFileExtensions(ColumnMetadata columnMetadata)
IndexType.BuildLifecycle	getIndexBuildLifecycle()
Class<DictionaryIndexConfig>	getIndexConfigClass()
String	getPrettyName()
protected void	handleIndexSpecificCleanup(TableConfig tableConfig)
static boolean	ignoreDictionaryOverride(boolean optimizeDictionary, boolean optimizeDictionaryForMetrics, double noDictionarySizeRatioThreshold, FieldSpec fieldSpec, FieldIndexConfigs fieldIndexConfigs, int cardinality, int totalNumberOfEntries)	This function evaluates whether to override dictionary (i.e use noDictionary) for a column even when its explicitly configured.
static Dictionary	read(PinotDataBuffer dataBuffer, ColumnMetadata metadata, DictionaryIndexConfig indexConfig)
static Dictionary	read(PinotDataBuffer dataBuffer, ColumnMetadata metadata, DictionaryIndexConfig indexConfig, String internIdentifierStr)
static Dictionary	read(SegmentDirectory.Reader segmentReader, ColumnMetadata columnMetadata)
static boolean	shouldUseVarLengthDictionary(String columnName, Set<String> varLengthDictColumns, FieldSpec.DataType columnStoredType, ColumnStatistics columnProfile)
boolean	shouldUseVarLengthDictionary(IndexCreationContext context, DictionaryIndexConfig indexConfig)
static boolean	shouldUseVarLengthDictionary(FieldSpec.DataType columnStoredType, ColumnStatistics profile)

Methods inherited from class org.apache.pinot.segment.spi.index.AbstractIndexType

convertToNewFormat, equals, getConfig, getId, getReaderFactory, hashCode, toString

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.pinot.segment.spi.index.IndexType

createMutableIndex, getIndexReader

Constructor Detail

DictionaryIndexType

protected DictionaryIndexType()

Method Detail

getIndexConfigClass

public Class<DictionaryIndexConfig> getIndexConfigClass()

Specified by:

getIndexConfigClass in interface IndexType<DictionaryIndexConfig,Dictionary,SegmentDictionaryCreator>

fromIndexLoadingConfig

public Map<String,DictionaryIndexConfig> fromIndexLoadingConfig(IndexLoadingConfig indexLoadingConfig)

Description copied from interface: ConfigurableFromIndexLoadingConfig

Returns a map that can be used to get the index config. This map is used with higher priority whenever the index configuration needs to be read from an IndexLoadingConfig. Sometimes IndexLoadingConfig is not completely configured and IndexLoadingConfig.getAllKnownColumns() does not return all columns in the table. Therefore the returned map may not have an entry for each column in the actual schema.

Specified by:

fromIndexLoadingConfig in interface ConfigurableFromIndexLoadingConfig<DictionaryIndexConfig>

Returns:

a map whose keys are the column names and the values are the index configuration for that column.

getDefaultConfig

public DictionaryIndexConfig getDefaultConfig()

Specified by:

getDefaultConfig in interface IndexType<DictionaryIndexConfig,Dictionary,SegmentDictionaryCreator>

getPrettyName

public String getPrettyName()

Specified by:

getPrettyName in interface IndexType<DictionaryIndexConfig,Dictionary,SegmentDictionaryCreator>

createDeserializer

public ColumnConfigDeserializer<DictionaryIndexConfig> createDeserializer()

Specified by:

createDeserializer in class AbstractIndexType<DictionaryIndexConfig,Dictionary,SegmentDictionaryCreator>

createIndexCreator

public SegmentDictionaryCreator createIndexCreator(IndexCreationContext context,
                                                   DictionaryIndexConfig indexConfig)

Specified by:

createIndexCreator in interface IndexType<DictionaryIndexConfig,Dictionary,SegmentDictionaryCreator>

shouldUseVarLengthDictionary

public boolean shouldUseVarLengthDictionary(IndexCreationContext context,
                                            DictionaryIndexConfig indexConfig)

shouldUseVarLengthDictionary

public static boolean shouldUseVarLengthDictionary(String columnName,
                                                   Set<String> varLengthDictColumns,
                                                   FieldSpec.DataType columnStoredType,
                                                   ColumnStatistics columnProfile)

shouldUseVarLengthDictionary

public static boolean shouldUseVarLengthDictionary(FieldSpec.DataType columnStoredType,
                                                   ColumnStatistics profile)

ignoreDictionaryOverride

public static boolean ignoreDictionaryOverride(boolean optimizeDictionary,
                                               boolean optimizeDictionaryForMetrics,
                                               double noDictionarySizeRatioThreshold,
                                               FieldSpec fieldSpec,
                                               FieldIndexConfigs fieldIndexConfigs,
                                               int cardinality,
                                               int totalNumberOfEntries)

This function evaluates whether to override dictionary (i.e use noDictionary) for a column even when its explicitly configured. This evaluation is for both dimension and metric column types.

createIndexCreator

public SegmentDictionaryCreator createIndexCreator(FieldSpec fieldSpec,
                                                   File indexDir,
                                                   boolean useVarLengthDictionary)
                                            throws Exception

Throws:

Exception

read

public static Dictionary read(SegmentDirectory.Reader segmentReader,
                              ColumnMetadata columnMetadata)
                       throws IOException

Throws:

IOException

read

public static Dictionary read(PinotDataBuffer dataBuffer,
                              ColumnMetadata metadata,
                              DictionaryIndexConfig indexConfig)
                       throws IOException

Throws:

IOException

read

public static Dictionary read(PinotDataBuffer dataBuffer,
                              ColumnMetadata metadata,
                              DictionaryIndexConfig indexConfig,
                              String internIdentifierStr)
                       throws IOException

Throws:

IOException

createReaderFactory

protected IndexReaderFactory<Dictionary> createReaderFactory()

Specified by:

createReaderFactory in class AbstractIndexType<DictionaryIndexConfig,Dictionary,SegmentDictionaryCreator>

createIndexHandler

public IndexHandler createIndexHandler(SegmentDirectory segmentDirectory,
                                       Map<String,FieldIndexConfigs> configsByCol,
                                       @Nullable
                                       Schema schema,
                                       @Nullable
                                       TableConfig tableConfig)

Specified by:

createIndexHandler in interface IndexType<DictionaryIndexConfig,Dictionary,SegmentDictionaryCreator>

getFileExtension

public static String getFileExtension()

getFileExtensions

public List<String> getFileExtensions(@Nullable
                                      ColumnMetadata columnMetadata)

Specified by:

getFileExtensions in interface IndexType<DictionaryIndexConfig,Dictionary,SegmentDictionaryCreator>

handleIndexSpecificCleanup

protected void handleIndexSpecificCleanup(TableConfig tableConfig)

Overrides:

handleIndexSpecificCleanup in class AbstractIndexType<DictionaryIndexConfig,Dictionary,SegmentDictionaryCreator>

createMutableDictionary

@Nullable
public static MutableDictionary createMutableDictionary(MutableIndexContext context,
                                                        DictionaryIndexConfig config)

Creates a MutableDictionary. Unlikes most indexes, while dictionaries are important when mutable segments are created, they do not follow the MutableIndex interface and therefore IndexType.createMutableIndex(MutableIndexContext, IndexConfig) is not implemented. This also means that dictionaries cannot be overridden in realtime tables.

getIndexBuildLifecycle

public IndexType.BuildLifecycle getIndexBuildLifecycle()

Specified by:

getIndexBuildLifecycle in interface IndexType<DictionaryIndexConfig,Dictionary,SegmentDictionaryCreator>