Package org.apache.pinot.segment.local.segment.creator.impl.fwd

Class MultiValueVarByteRawIndexCreator

java.lang.Object

org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueVarByteRawIndexCreator

All Implemented Interfaces:

Closeable, AutoCloseable, ForwardIndexCreator, IndexCreator

public class MultiValueVarByteRawIndexCreator
extends Object
implements ForwardIndexCreator

Forward index creator for raw (non-dictionary-encoded) single-value column of variable length data type (STRING, BYTES).

Constructor Summary

Constructors

Constructor	Description
MultiValueVarByteRawIndexCreator(File baseIndexDir, ChunkCompressionType compressionType, String column, int totalDocs, FieldSpec.DataType valueType, int maxRowLengthInBytes, int maxNumberOfElements)	Create a var-byte raw index creator for the given column
MultiValueVarByteRawIndexCreator(File baseIndexDir, ChunkCompressionType compressionType, String column, int totalDocs, FieldSpec.DataType valueType, int writerVersion, int maxRowLengthInBytes, int maxNumberOfElements)	Create a var-byte raw index creator for the given column

Method Summary

Modifier and Type	Method	Description
void	close()
static int	getElementLengthStoragePrefixInBytes(int maxNumberOfElements)	The actual content in an MV array is prepended with 2 prefixes: 1.
static int	getMaxRowDataLengthInBytes(int totalMaxLength, int maxNumberOfElements)	The actual content in an MV array is prepended with 2 prefixes: 1.
static int	getNumElementsStoragePrefix()	The actual content in an MV array is prepended with 2 prefixes: 1.
static int	getTotalRowStorageBytes(int maxNumberOfElements, int maxRowDataLengthInBytes)	The actual content in an MV array is prepended with 2 prefixes: 1.
FieldSpec.DataType	getValueType()
boolean	isDictionaryEncoded()
boolean	isSingleValue()
void	putBytesMV(byte[][] values)
void	putStringMV(String[] values)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator

add, add, putBigDecimal, putBytes, putDictId, putDictIdMV, putDouble, putDoubleMV, putFloat, putFloatMV, putInt, putIntMV, putLong, putLongMV, putString, seal

Constructor Detail

MultiValueVarByteRawIndexCreator

public MultiValueVarByteRawIndexCreator(File baseIndexDir,
                                        ChunkCompressionType compressionType,
                                        String column,
                                        int totalDocs,
                                        FieldSpec.DataType valueType,
                                        int maxRowLengthInBytes,
                                        int maxNumberOfElements)
                                 throws IOException

Create a var-byte raw index creator for the given column

Parameters:

baseIndexDir - Index directory

compressionType - Type of compression to use

column - Name of column to index

totalDocs - Total number of documents to index

valueType - Type of the values

maxRowLengthInBytes - the length in bytes of the largest row

maxNumberOfElements - the maximum number of elements in a row

Throws:

IOException

MultiValueVarByteRawIndexCreator

public MultiValueVarByteRawIndexCreator(File baseIndexDir,
                                        ChunkCompressionType compressionType,
                                        String column,
                                        int totalDocs,
                                        FieldSpec.DataType valueType,
                                        int writerVersion,
                                        int maxRowLengthInBytes,
                                        int maxNumberOfElements)
                                 throws IOException

Create a var-byte raw index creator for the given column

Parameters:

baseIndexDir - Index directory

compressionType - Type of compression to use

column - Name of column to index

totalDocs - Total number of documents to index

valueType - Type of the values

maxRowLengthInBytes - the size in bytes of the largest row, the chunk size cannot be smaller than this

maxNumberOfElements - the maximum number of elements in a row

writerVersion - writer format version

Throws:

IOException

Method Detail

isDictionaryEncoded

public boolean isDictionaryEncoded()

Specified by:

isDictionaryEncoded in interface ForwardIndexCreator

isSingleValue

public boolean isSingleValue()

Specified by:

isSingleValue in interface ForwardIndexCreator

getValueType

public FieldSpec.DataType getValueType()

Specified by:

getValueType in interface ForwardIndexCreator

putStringMV

public void putStringMV(String[] values)

Specified by:

putStringMV in interface ForwardIndexCreator

putBytesMV

public void putBytesMV(byte[][] values)

Specified by:

putBytesMV in interface ForwardIndexCreator

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Throws:

IOException

getElementLengthStoragePrefixInBytes

public static int getElementLengthStoragePrefixInBytes(int maxNumberOfElements)

The actual content in an MV array is prepended with 2 prefixes: 1. elementLengthStoragePrefixInBytes - bytes required to store the length of each element in the largest array 2. numElementsStoragePrefixInBytes - Number of elements in the array This function returns the total bytes needed to store (1) elementLengthStoragePrefixInBytes

getNumElementsStoragePrefix

public static int getNumElementsStoragePrefix()

The actual content in an MV array is prepended with 2 prefixes: 1. elementLengthStoragePrefixInBytes - bytes required to store the length of each element in the largest array 2. numElementsStoragePrefixInBytes - Number of elements in the array This function returns the bytes needed to store (2) numElementsStoragePrefixInBytes

getTotalRowStorageBytes

public static int getTotalRowStorageBytes(int maxNumberOfElements,
                                          int maxRowDataLengthInBytes)

The actual content in an MV array is prepended with 2 prefixes: 1. elementLengthStoragePrefixInBytes - bytes required to store the length of each element in the largest array 2. numElementsStoragePrefixInBytes - Number of elements in the array This function returns the bytes needed to store the (1), (2) and the actual content.

getMaxRowDataLengthInBytes

public static int getMaxRowDataLengthInBytes(int totalMaxLength,
                                             int maxNumberOfElements)

The actual content in an MV array is prepended with 2 prefixes: 1. elementLengthStoragePrefixInBytes - bytes required to store the length of each element in the largest array 2. numberOfElementsStoragePrefix - Number of elements in the array This function returns the bytes needed to store the actual content.