com.topologi.diffx.load.text

Class TokenizerByWord

java.lang.Object

com.topologi.diffx.load.text.TokenizerByWord

All Implemented Interfaces:

TextTokenizer

public final class TokenizerByWord
extends Object
implements TextTokenizer

The tokeniser for characters events.

This class is not synchronized.

Version:

11 May 2010

Author:

Christophe Lauret

Constructor Summary

Constructors

Constructor and Description
TokenizerByWord(WhiteSpaceProcessing whitespace) Creates a new tokenizer.

Method Summary

Modifier and Type	Method and Description
TextGranularity	granurality() Always TextGranularity.WORD.
List<TextEvent>	tokenize(CharSequence seq) Returns the list of TextEvent corresponding to the specified character sequence.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TokenizerByWord

public TokenizerByWord(WhiteSpaceProcessing whitespace)

Creates a new tokenizer.

Parameters:

whitespace - the whitespace processing for this tokenizer.

Throws:

NullPointerException - if the white space processing is not specified.

Method Detail

tokenize

public List<TextEvent> tokenize(CharSequence seq)

Returns the list of TextEvent corresponding to the specified character sequence.

Specified by:

tokenize in interface TextTokenizer

Parameters:

seq - the character sequence to tokenize.

Returns:

the corresponding list.

granurality

public TextGranularity granurality()

Always TextGranularity.WORD. Returns the text granularity of this tokenizer.

Specified by:

granurality in interface TextTokenizer

Returns:

the text granularity of this tokenizer.