com.wcohen.ss.api

Interface SourcedTokenizer

All Superinterfaces:

Tokenizer

All Known Implementing Classes:

SimpleSourcedTokenizer

public interface SourcedTokenizer
extends Tokenizer

Split a string into tokens, retaining provinance.

Method Summary

Modifier and Type	Method and Description
SourcedToken[]	sourcedTokenize(String input, String source) Return tokenized version of a string, as an array of SourcedToken objects.

Methods inherited from interface com.wcohen.ss.api.Tokenizer

intern, maxTokenIndex, tokenIterator, tokenize

Method Detail

sourcedTokenize

SourcedToken[] sourcedTokenize(String input,
                               String source)

Return tokenized version of a string, as an array of SourcedToken objects.