org.languagetool.tokenizers.crh

Class CrimeanTatarWordTokenizer

java.lang.Object

org.languagetool.tokenizers.WordTokenizer

org.languagetool.tokenizers.crh.CrimeanTatarWordTokenizer

All Implemented Interfaces:

Tokenizer

public class CrimeanTatarWordTokenizer
extends WordTokenizer

Field Summary

Fields inherited from class org.languagetool.tokenizers.WordTokenizer

REMOVED_EMOJI

Constructor Summary

Constructors

Constructor and Description
CrimeanTatarWordTokenizer()

Method Summary

Modifier and Type	Method and Description
String	getTokenizingCharacters()
List<String>	tokenize(String text) Tokenizes text.

Methods inherited from class org.languagetool.tokenizers.WordTokenizer

getProtocols, isCurrencyExpression, isEMail, isUrl, joinEMails, joinEMailsAndUrls, joinUrls, replaceEmojis, restoreEmojis, splitCurrencyExpression

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CrimeanTatarWordTokenizer

public CrimeanTatarWordTokenizer()

Method Detail

getTokenizingCharacters

public String getTokenizingCharacters()

Overrides:

getTokenizingCharacters in class WordTokenizer

tokenize

public List<String> tokenize(String text)

Tokenizes text. The CrimeanTatar tokenizer differs from the standard one in two respects:

1. it does not treat the hyphen as part of the word if the hyphen is at the end of the word;
2. it includes n-dash as a tokenizing character, as it is used without a whitespace in CrimeanTatar.

Specified by:

tokenize in interface Tokenizer

Overrides:

tokenize in class WordTokenizer

Parameters:

text - String of words to tokenize.