Package edu.mit.simile.vicino

Class NGramTokenizer

java.lang.Object

edu.mit.simile.vicino.NGramTokenizer

All Implemented Interfaces:

com.wcohen.ss.api.Tokenizer

public class NGramTokenizer
extends Object
implements com.wcohen.ss.api.Tokenizer

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
class	NGramTokenizer.BasicToken

Constructor Summary

Constructors

Constructor	Description
NGramTokenizer(int ngram_size)

Method Summary

Modifier and Type	Method	Description
com.wcohen.ss.api.Token	intern(String s)
int	maxTokenIndex()
Iterator<com.wcohen.ss.api.Token>	tokenIterator()
com.wcohen.ss.api.Token[]	tokenize(String str)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

NGramTokenizer

public NGramTokenizer(int ngram_size)

Method Details

tokenize

public com.wcohen.ss.api.Token[] tokenize(String str)

Specified by:

tokenize in interface com.wcohen.ss.api.Tokenizer

intern

public com.wcohen.ss.api.Token intern(String s)

Specified by:

intern in interface com.wcohen.ss.api.Tokenizer

tokenIterator

public Iterator<com.wcohen.ss.api.Token> tokenIterator()

Specified by:

tokenIterator in interface com.wcohen.ss.api.Tokenizer

maxTokenIndex

public int maxTokenIndex()

Specified by:

maxTokenIndex in interface com.wcohen.ss.api.Tokenizer