Package org.apache.pinot.segment.local.segment.creator.impl.inv

Class OffHeapBitmapInvertedIndexCreator

  • All Implemented Interfaces:
    Closeable, AutoCloseable, DictionaryBasedInvertedIndexCreator, InvertedIndexCreator
    public final class OffHeapBitmapInvertedIndexCreator
extends Object
implements DictionaryBasedInvertedIndexCreator
    Implementation of DictionaryBasedInvertedIndexCreator that uses off-heap memory.

    We use 2 passes to create the inverted index.

    • In the first pass (adding values phase), when add() method is called, store the dictIds into the forward index value buffer (for multi-valued column also store number of values for each docId into forward index length buffer). We also compute the inverted index length for each dictId while adding values.
    • In the second pass (processing values phase), when seal() method is called, all the dictIds should already been added. We first reorder the values into the inverted index buffers by going over the dictIds in forward index value buffer (for multi-valued column we also need forward index length buffer to get the docId for each dictId).

      Once we have the inverted index buffers, we simply go over them and create the bitmap for each dictId and serialize them into a file.

    Based on the number of values we need to store, we use direct memory or MMap file to allocate the buffer.