Package com.wcohen.ss.tokens

Class Summary

Class	Description
BasicSourcedToken	An interned version of a string, with provinance information
BasicToken	An interned version of a string.
CharacterTokenizer	Character tokenizer implementation.
NGramTokenizer	Wraps another tokenizer, and adds all computes all ngrams of characters from a single token produced by the inner tokenizer.
SimpleSourcedTokenizer	Simple implementation of a Tokenizer.
SimpleTokenizer	Simple implementation of a Tokenizer.
TagLinkToken