Uses of Enum
org.tribuo.util.tokens.Token.TokenType

Packages that use Token.TokenType

Package	Description
org.tribuo.util.tokens	Core definitions for tokenization.
org.tribuo.util.tokens.impl	Simple fixed rule tokenizers.
org.tribuo.util.tokens.impl.wordpiece	Provides an implementation of a Wordpiece tokenizer which implements to the Tribuo Tokenizer API.
org.tribuo.util.tokens.universal	An implementation of a "universal" tokenizer which will split on word boundaries or character boundaries for languages where word boundaries are contextual.

Uses of Token.TokenType in org.tribuo.util.tokens

Fields in org.tribuo.util.tokens declared as Token.TokenType

Modifier and Type	Field	Description
final Token.TokenType	Token.type	The token type.

Methods in org.tribuo.util.tokens that return Token.TokenType

Modifier and Type	Method	Description
Token.TokenType	Tokenizer.getType()	Gets the type of the current token.
static Token.TokenType	Token.TokenType.valueOf(String name)	Returns the enum constant of this type with the specified name.
static Token.TokenType[]	Token.TokenType.values()	Returns an array containing the constants of this enum type, in the order they are declared.

Constructors in org.tribuo.util.tokens with parameters of type Token.TokenType

Modifier	Constructor	Description
	Token(String text, int start, int end, Token.TokenType type)	Constructs a token.

Uses of Token.TokenType in org.tribuo.util.tokens.impl

Fields in org.tribuo.util.tokens.impl declared as Token.TokenType

Modifier and Type	Field	Description
final Token.TokenType	SplitFunctionTokenizer.SplitResult.tokenType	The token type.

Methods in org.tribuo.util.tokens.impl that return Token.TokenType

Modifier and Type	Method	Description
Token.TokenType	BreakIteratorTokenizer.getType()
Token.TokenType	NonTokenizer.getType()
Token.TokenType	ShapeTokenizer.getType()
Token.TokenType	SplitFunctionTokenizer.getType()
Token.TokenType	SplitPatternTokenizer.getType()

Uses of Token.TokenType in org.tribuo.util.tokens.impl.wordpiece

Methods in org.tribuo.util.tokens.impl.wordpiece that return Token.TokenType

Modifier and Type	Method	Description
Token.TokenType	WordpieceTokenizer.getType()

Uses of Token.TokenType in org.tribuo.util.tokens.universal

Fields in org.tribuo.util.tokens.universal declared as Token.TokenType

Modifier and Type	Field	Description
Token.TokenType	Range.type	The current token type.

Methods in org.tribuo.util.tokens.universal that return Token.TokenType

Modifier and Type	Method	Description
Token.TokenType	UniversalTokenizer.getType()

Methods in org.tribuo.util.tokens.universal with parameters of type Token.TokenType

Modifier and Type	Method	Description
void	Range.setType(Token.TokenType type)	Sets the token type.