Package org.apache.pinot.segment.local.segment.creator.impl.text

Class LuceneTextIndexCreator

java.lang.Object

org.apache.pinot.segment.local.segment.index.text.AbstractTextIndexCreator

org.apache.pinot.segment.local.segment.creator.impl.text.LuceneTextIndexCreator

All Implemented Interfaces:

Closeable, AutoCloseable, TextIndexCreator, IndexCreator

public class LuceneTextIndexCreator
extends AbstractTextIndexCreator

This is used to create Lucene based text index. Used for both offline from SegmentColumnarIndexCreator and realtime from RealtimeLuceneTextIndex

Field Summary

Fields

Modifier and Type	Field	Description
static org.apache.lucene.analysis.CharArraySet	ENGLISH_STOP_WORDS_SET
static String	LUCENE_INDEX_DOC_ID_COLUMN_NAME

Constructor Summary

Constructors

Constructor	Description
LuceneTextIndexCreator(String column, File segmentIndexDir, boolean commit, List<String> stopWordsInclude, List<String> stopWordsExclude, boolean useCompoundFile, int maxBufferSizeMB)	Called by SegmentColumnarIndexCreator when building an offline segment.
LuceneTextIndexCreator(IndexCreationContext context, TextIndexConfig indexConfig)

Method Summary

Modifier and Type	Method	Description
void	add(String document)
void	add(String[] documents, int length)
void	close()
static HashSet<String>	getDefaultEnglishStopWordsSet()
org.apache.lucene.index.IndexWriter	getIndexWriter()
void	seal()

Methods inherited from class org.apache.pinot.segment.local.segment.index.text.AbstractTextIndexCreator

add, add

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LUCENE_INDEX_DOC_ID_COLUMN_NAME

public static final String LUCENE_INDEX_DOC_ID_COLUMN_NAME

See Also:

Constant Field Values

ENGLISH_STOP_WORDS_SET

public static final org.apache.lucene.analysis.CharArraySet ENGLISH_STOP_WORDS_SET

Constructor Detail

LuceneTextIndexCreator

public LuceneTextIndexCreator(String column,
                              File segmentIndexDir,
                              boolean commit,
                              @Nullable
                              List<String> stopWordsInclude,
                              @Nullable
                              List<String> stopWordsExclude,
                              boolean useCompoundFile,
                              int maxBufferSizeMB)

Called by SegmentColumnarIndexCreator when building an offline segment. Similar to how it creates per column dictionary, forward and inverted index, a text index is also created if text search is enabled on a column.

Parameters:

column - column name

segmentIndexDir - segment index directory

commit - true if the index should be committed (at the end after all documents have been added), false if index should not be committed Note on commit: Once SegmentColumnarIndexCreator finishes indexing all documents/rows for the segment, we need to commit and close the Lucene index which will internally persist the index on disk, do the necessary resource cleanup etc. We commit during InvertedIndexCreator.seal() and close during Closeable.close(). This lucene index writer is used by both offline and realtime (both during indexing in-memory MutableSegment and later during conversion to offline). Since realtime segment conversion is again going to go through the offline indexing path and will do everything (indexing, commit, close etc), there is no need to commit the index from the realtime side. So when the realtime segment is destroyed (which is after the realtime segment has been committed and converted to offline), we close this lucene index writer to release resources but don't commit.

stopWordsInclude - the words to include in addition to the default stop word list

stopWordsExclude - the words to exclude from the default stop word list

LuceneTextIndexCreator

public LuceneTextIndexCreator(IndexCreationContext context,
                              TextIndexConfig indexConfig)

Method Detail

getDefaultEnglishStopWordsSet

public static HashSet<String> getDefaultEnglishStopWordsSet()

getIndexWriter

public org.apache.lucene.index.IndexWriter getIndexWriter()

add

public void add(String document)

add

public void add(String[] documents,
                int length)

seal

public void seal()

close

public void close()
           throws IOException

Throws:

IOException