TextTokenizer (Nitrite 4.2.0 API)

All Known Implementing Classes:

BaseTextTokenizer, EnglishTextTokenizer, UniversalTextTokenizer
```
public interface TextTokenizer
```
An abstract class representing a stop-word based text tokenizer.

Since:

1.0

Author:

Anindya Chatterjee.

See Also:

EnglishTextTokenizer

Method Summary

All Methods Instance Methods Abstract Methods
Modifier and Type	Method	Description
`Languages`	`getLanguage()`	Gets the language for the tokenizer.
`Set<String>`	`stopWords()`	Gets all stop-words for a language.
`Set<String>`	`tokenize(String text)`	Tokenize a `text` and discards all stop-words from it.

- Method Detail
  - getLanguage
```
Languages getLanguage()
```
    Gets the language for the tokenizer.
    
    Returns:
    
    the language for this tokenizer.
  - tokenize
```
Set<String> tokenize(String text)
```
    Tokenize a text and discards all stop-words from it.
    
    Parameters:
    
    text - the text to tokenize
    
    Returns:
    
    the set of tokens.
  - stopWords
```
Set<String> stopWords()
```
    Gets all stop-words for a language.
    
    Returns:
    
    the set of all stop-words.