Package org.apache.pinot.segment.local.segment.creator.impl.inv.json

Class OffHeapJsonIndexCreator

java.lang.Object

org.apache.pinot.segment.local.segment.creator.impl.inv.json.BaseJsonIndexCreator

org.apache.pinot.segment.local.segment.creator.impl.inv.json.OffHeapJsonIndexCreator

All Implemented Interfaces:

Closeable, AutoCloseable, JsonIndexCreator, IndexCreator

public class OffHeapJsonIndexCreator
extends BaseJsonIndexCreator

Implementation of JsonIndexCreator that uses off-heap memory.

The posting lists (map from value to doc ids) are initially stored in a TreeMap, then flushed into a file for every 100,000 documents (unflattened records) added. After all the documents are added, we read all the posting lists from the file and merge them using a priority queue to calculate the final posting lists. Then we generate the string dictionary and inverted index from the final posting lists and create the json index on top of them.

Off-heap creator uses less heap memory, but is more expensive on computation and needs flush data to disk which can slow down the creation because of the IO latency. Use off-heap creator in the environment where there is limited heap memory or garbage collection can cause performance issue (e.g. index creation at loading time on Pinot Server).

Field Summary

Fields inherited from class org.apache.pinot.segment.local.segment.creator.impl.inv.json.BaseJsonIndexCreator

HEADER_LENGTH, VERSION_1, VERSION_2

Fields inherited from interface org.apache.pinot.segment.spi.index.creator.JsonIndexCreator

KEY_VALUE_SEPARATOR

Constructor Summary

Constructors

Constructor	Description
OffHeapJsonIndexCreator(File indexDir, String columnName, JsonIndexConfig jsonIndexConfig)

Method Summary

Modifier and Type	Method	Description
void	seal()

Methods inherited from class org.apache.pinot.segment.local.segment.creator.impl.inv.json.BaseJsonIndexCreator

add, close

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.pinot.segment.spi.index.creator.JsonIndexCreator

add, add

Constructor Detail

OffHeapJsonIndexCreator

public OffHeapJsonIndexCreator(File indexDir,
                               String columnName,
                               JsonIndexConfig jsonIndexConfig)
                        throws IOException

Throws:

IOException

Method Detail

seal

public void seal()
          throws IOException

Throws:

IOException