Package org.apache.lucene.analysis.util

Class TokenizerFactory

java.lang.Object

org.apache.lucene.analysis.util.AbstractAnalysisFactory

org.apache.lucene.analysis.util.TokenizerFactory

Direct Known Subclasses:

ArabicLetterTokenizerFactory, ChineseTokenizerFactory, CJKTokenizerFactory, ClassicTokenizerFactory, EdgeNGramTokenizerFactory, KeywordTokenizerFactory, LetterTokenizerFactory, LowerCaseTokenizerFactory, NGramTokenizerFactory, PathHierarchyTokenizerFactory, PatternTokenizerFactory, RussianLetterTokenizerFactory, StandardTokenizerFactory, UAX29URLEmailTokenizerFactory, WhitespaceTokenizerFactory, WikipediaTokenizerFactory

public abstract class TokenizerFactory
extends AbstractAnalysisFactory

Abstract parent class for analysis factories that create Tokenizer instances.

Field Summary

Fields inherited from class org.apache.lucene.analysis.util.AbstractAnalysisFactory

LUCENE_MATCH_VERSION_PARAM

Method Summary

Modifier and Type	Method	Description
static Set<String>	availableTokenizers()	returns a list of all available tokenizer names from context classpath
final Tokenizer	create(Reader input)	Creates a TokenStream of the specified input using the default attribute factory.
abstract Tokenizer	create(AttributeSource.AttributeFactory factory, Reader input)	Creates a TokenStream of the specified input using the given AttributeFactory
static TokenizerFactory	forName(String name, Map<String,String> args)	looks up a tokenizer by name from context classpath
static Class<? extends TokenizerFactory>	lookupClass(String name)	looks up a tokenizer class by name from context classpath
static void	reloadTokenizers(ClassLoader classloader)	Reloads the factory list from the given ClassLoader.

Methods inherited from class org.apache.lucene.analysis.util.AbstractAnalysisFactory

get, get, get, get, get, getChar, getClassArg, getLuceneMatchVersion, getOriginalArgs, getSet, isExplicitLuceneMatchVersion, require, require, require, requireChar, setExplicitLuceneMatchVersion

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

forName

public static TokenizerFactory forName(String name,
 Map<String,String> args)

looks up a tokenizer by name from context classpath

lookupClass

public static Class<? extends TokenizerFactory> lookupClass(String name)

looks up a tokenizer class by name from context classpath

availableTokenizers

public static Set<String> availableTokenizers()

returns a list of all available tokenizer names from context classpath

reloadTokenizers

public static void reloadTokenizers(ClassLoader classloader)

Reloads the factory list from the given ClassLoader. Changes to the factories are visible after the method ends, all iterators (availableTokenizers(),...) stay consistent.

NOTE: Only new factories are added, existing ones are never removed or replaced.

This method is expensive and should only be called for discovery of new factories on the given classpath/classloader!

create

public final Tokenizer create(Reader input)

Creates a TokenStream of the specified input using the default attribute factory.

create

public abstract Tokenizer create(AttributeSource.AttributeFactory factory,
 Reader input)

Creates a TokenStream of the specified input using the given AttributeFactory