Package org.apache.pinot.segment.local.realtime.impl.forward

Class FixedByteMVMutableForwardIndex

  • All Implemented Interfaces:
    Closeable, AutoCloseable, MutableForwardIndex, ForwardIndexReader<ForwardIndexReaderContext>
    public class FixedByteMVMutableForwardIndex
extends Object
implements MutableForwardIndex
    This class provides expandable off-heap implementation to store a multi-valued column across a number of rows. The maximum number of values in any row must be known while invoking the constructor. Other than that, this class allocates additional memory as needed to accommodate any number of rows. Writes into the data structure are strictly sequential, but reads can be random. Writes are of type: setIntArray(int rowNumber, int[] values) It is expected that rowNumber starts with 0 and increments by 1 on each invocation, and that it is decided ahead of time that the class is used to store a certain type of data structure (int arrays, or char arrays, etc.) Mix & match is not allowed. Two kinds of data structures are used in this class. 1. A header (essentially an index into the other data structure) that has one entry per row. The entry has 3 integers - data buffer ID - offset in the data buffer where column values start - length (number of values in the multi-valued column). New header structures are added as new rows come in. Each header class holds the same number of rows (for easy lookup) 2. A data buffer that has the values for the column that the header points to. Data buffers are added as needed, whenever we reach a limitation that we cannot fit the values of a column in the current buffer. Note that data buffers and headers grow independently. Data format HEADER SECTION 0 bufferId startIndex length bufferId startIndex length bufferId startIndex length ... HEADER SECTION 1 bufferId startIndex length bufferId startIndex length bufferId startIndex length ... Data BUFFER SECTION 0 [set of values of row 0] [set of values of row 1] ..... [set of values of row m] Data BUFFER SECTION 1 [set of values of row m +1 ] [set of values of row M +2] ..... [set of values of row ] Data BUFFER SECTION N [set of values of row ... ] [set of values of row ...] ..... [set of values of row n]