com.wcohen.ss.tokens

Class CharacterTokenizer

java.lang.Object

com.wcohen.ss.tokens.CharacterTokenizer

All Implemented Interfaces:

Tokenizer

public class CharacterTokenizer
extends Object
implements Tokenizer

Character tokenizer implementation. Tokens are single characters of the source string.

Field Summary

Fields

Modifier and Type	Field and Description
static CharacterTokenizer	DEFAULT_TOKENIZER

Constructor Summary

Constructors

Constructor and Description
CharacterTokenizer(boolean ignorePunctuation, boolean ignoreCase)

Method Summary

Modifier and Type	Method and Description
Token	intern(String s) Convert a given string into a token.
static void	main(String[] argv) Test routine
int	maxTokenIndex() Return the higest index of any interned token
void	setIgnoreCase(boolean flag)
void	setIgnorePunctuation(boolean flag)
Iterator<Token>	tokenIterator() Return an iterator over interned tokens
Token[]	tokenize(String input) Return tokenized version of a string.
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_TOKENIZER

public static final CharacterTokenizer DEFAULT_TOKENIZER

Constructor Detail

CharacterTokenizer

public CharacterTokenizer(boolean ignorePunctuation,
                          boolean ignoreCase)

Method Detail

setIgnorePunctuation

public void setIgnorePunctuation(boolean flag)

setIgnoreCase

public void setIgnoreCase(boolean flag)

toString

public String toString()

Overrides:

toString in class Object

tokenize

public Token[] tokenize(String input)

Return tokenized version of a string. Tokens are sequences of alphanumerics, or any single punctuation character.

Specified by:

tokenize in interface Tokenizer

intern

public Token intern(String s)

Description copied from interface: Tokenizer

Convert a given string into a token. The intern function should have these properties: (1) If s1.equals(s2), then intern(s1)==intern(s2). (2) If no string equal to s1 has ever been interned before, then intern(s1).getIndex() will be larger than every previously-assigned index--i.e, token 'indexes' are assigned in increasing order.

Specified by:

intern in interface Tokenizer

tokenIterator

public Iterator<Token> tokenIterator()

Description copied from interface: Tokenizer

Return an iterator over interned tokens

Specified by:

tokenIterator in interface Tokenizer

maxTokenIndex

public int maxTokenIndex()

Description copied from interface: Tokenizer

Return the higest index of any interned token

Specified by:

maxTokenIndex in interface Tokenizer

main

public static void main(String[] argv)

Test routine