Uses of Class
org.tribuo.util.tokens.Token

Packages that use Token

Package	Description
org.tribuo.util.tokens	Core definitions for tokenization.
org.tribuo.util.tokens.impl.wordpiece	Provides an implementation of a Wordpiece tokenizer which implements to the Tribuo Tokenizer API.

Uses of Token in org.tribuo.util.tokens

Methods in org.tribuo.util.tokens that return Token

Modifier and Type	Method	Description
default Token	Tokenizer.getToken()	Generates a Token object from the current state of the tokenizer.

Methods in org.tribuo.util.tokens that return types with arguments of type Token

Modifier and Type	Method	Description
default List<Token>	Tokenizer.tokenize(CharSequence cs)	Uses this tokenizer to tokenize a string and return the list of tokens that were generated.

Uses of Token in org.tribuo.util.tokens.impl.wordpiece

Methods in org.tribuo.util.tokens.impl.wordpiece that return Token

Modifier and Type	Method	Description
Token	WordpieceTokenizer.getToken()