Package opennlp.tools.tokenize

Class TokenizerModel

java.lang.Object

opennlp.tools.util.model.BaseModel

opennlp.tools.tokenize.TokenizerModel

All Implemented Interfaces:

Serializable, ArtifactProvider

public final class TokenizerModel
extends BaseModel

The TokenizerModel is the model used by a learnable Tokenizer.

See Also:

• TokenizerME
• Serialized Form

Field Summary

Fields inherited from class opennlp.tools.util.model.BaseModel

TRAINING_CUTOFF_PROPERTY, TRAINING_EVENTHASH_PROPERTY, TRAINING_ITERATIONS_PROPERTY

Constructor Summary

Constructors

Constructor	Description
TokenizerModel(File modelFile)	Initializes the current instance.
TokenizerModel(InputStream in)	Initializes the current instance.
TokenizerModel(URL modelURL)	Initializes the current instance.
TokenizerModel(Path modelPath)
TokenizerModel(MaxentModel tokenizerModel, Map<String,String> manifestInfoEntries, TokenizerFactory tokenizerFactory)	Initializes the current instance.

Method Summary

Modifier and Type	Method	Description
Dictionary	getAbbreviations()
TokenizerFactory	getFactory()
MaxentModel	getMaxentModel()
boolean	useAlphaNumericOptimization()

Methods inherited from class opennlp.tools.util.model.BaseModel

getArtifact, getLanguage, getManifestProperty, getVersion, isLoadedFromSerialized, serialize, serialize, serialize

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

TokenizerModel

public TokenizerModel(MaxentModel tokenizerModel,
 Map<String,String> manifestInfoEntries,
 TokenizerFactory tokenizerFactory)

Initializes the current instance.

Parameters:

tokenizerModel - the model

manifestInfoEntries - the manifest

tokenizerFactory - the factory

TokenizerModel

public TokenizerModel(InputStream in)
               throws IOException

Initializes the current instance.

Parameters:

in - the Input Stream to load the model from

Throws:

IOException - if reading from the stream fails in anyway

InvalidFormatException - if the stream doesn't have the expected format

TokenizerModel

public TokenizerModel(File modelFile)
               throws IOException

Initializes the current instance.

Parameters:

modelFile - the file containing the tokenizer model

Throws:

IOException - if reading from the stream fails in anyway

TokenizerModel

public TokenizerModel(Path modelPath)
               throws IOException

Throws:

IOException

TokenizerModel

public TokenizerModel(URL modelURL)
               throws IOException

Initializes the current instance.

Parameters:

modelURL - the URL pointing to the tokenizer model

Throws:

IOException - if reading from the stream fails in anyway

Method Details

getFactory

public TokenizerFactory getFactory()

getMaxentModel

public MaxentModel getMaxentModel()

getAbbreviations

public Dictionary getAbbreviations()

useAlphaNumericOptimization

public boolean useAlphaNumericOptimization()