All Classes and Interfaces

Class	Description
BreakIteratorTokenizer	A tokenizer wrapping a BreakIterator instance.
BreakIteratorTokenizerOptions	CLI options for a BreakIteratorTokenizer.
CoreTokenizerOptions	CLI Options for all the tokenizers in the core package.
CoreTokenizerOptions.CoreTokenizerType	Tokenizer type.
NonTokenizer	A convenience class for when you are required to provide a tokenizer but you don't actually want to split up the text into tokens.
Range	A range currently being segmented.
ShapeTokenizer	This tokenizer is loosely based on the notion of word shape which is a common feature used in NLP.
SplitCharactersTokenizer	This implementation of Tokenizer is instantiated with an array of characters that are considered split characters.
SplitCharactersTokenizer.SplitCharactersSplitterFunction	Splits tokens at the supplied characters.
SplitCharactersTokenizerOptions	CLI options for a SplitCharactersTokenizer.
SplitFunctionTokenizer	This class supports character-by-character (that is, codepoint-by-codepoint) iteration over input text to create tokens.
SplitFunctionTokenizer.SplitFunction	An interface for checking if the text should be split at the supplied codepoint.
SplitFunctionTokenizer.SplitResult	A combination of a SplitFunctionTokenizer.SplitType and a Token.TokenType.
SplitFunctionTokenizer.SplitType	Defines different ways that a tokenizer can split the input text at a given character.
SplitPatternTokenizer	This implementation of Tokenizer is instantiated with a regular expression pattern which determines how to split a string into tokens.
SplitPatternTokenizerOptions	CLI options for a SplitPatternTokenizer.
Token	A single token extracted from a String.
Token.TokenType	Tokenizers may product multiple kinds of tokens, depending on the application to which they're being put.
TokenizationException	Wraps exceptions thrown by tokenizers.
Tokenizer	An interface for things that tokenize text: breaking it into words according to some set of rules.
TokenizerOptions	CLI Options for creating a tokenizer.
UniversalTokenizer	This class was originally written for the purpose of document indexing in an information retrieval context (principally used in Sun Labs' Minion search engine).
WhitespaceTokenizer	A simple tokenizer that splits on whitespace.
Wordpiece	This is vanilla implementation of the Wordpiece algorithm as found here: https://github.com/huggingface/transformers/blob/master/src/transformers/models/bert/tokenization_bert.py
WordpieceBasicTokenizer	This is a tokenizer that is used "upstream" of WordpieceTokenizer and implements much of the functionality of the 'BasicTokenizer' implementation in huggingface.
WordpieceTokenizer	This Tokenizer is meant to be a reasonable approximation of the BertTokenizer defined here.