de.danielnaber.jwordsplitter

Class AbstractWordSplitter

java.lang.Object

de.danielnaber.jwordsplitter.AbstractWordSplitter

Direct Known Subclasses:

GermanWordSplitter

public abstract class AbstractWordSplitter
extends Object

This class can split compound words into their smallest parts (atoms). For example "Erhebungsfehler" will be split into "erhebung" and "fehler", if "erhebung" and "fehler" are in the dictionary and "erhebungsfehler" is not. Thus how words are split only depends on the contents of the dictionary. A dictionary for German is included.

This is especially useful for German words but it will work with all languages. The order of the words in the collection will be identical to their appearance in the connected word. It's good to provide a large dictionary.

Please note: We don't expect to have any special chars here (!":;,.-_, etc.). Only a set of characters and only one word.

Author:

Daniel Naber

Field Summary

Fields

Modifier and Type	Field and Description
protected Set<String>	words

Constructor Summary

Constructors

Constructor and Description
AbstractWordSplitter(boolean hideInterfixCharacters) Create a word splitter that uses the embedded dictionary.
AbstractWordSplitter(boolean hideInterfixCharacters, File plainTextDict)
AbstractWordSplitter(boolean hideInterfixCharacters, InputStream plainTextDict)
AbstractWordSplitter(boolean hideInterfixCharacters, Set<String> words)

Method Summary

Modifier and Type	Method and Description
void	addException(String completeWord, List<String> wordParts)
List<List<String>>	getAllSplits(String word) Experimental: Split a word with unknown parts, typically because one part has a typo.
protected abstract int	getDefaultMinimumWordLength()
protected abstract de.danielnaber.jwordsplitter.GermanInterfixDisambiguator	getDisambiguator()
protected abstract Collection<String>	getInterfixCharacters() Interfix elements in lowercase, e.g. at least "s" for German.
List<String>	getSubWords(String word)
protected abstract Set<String>	getWordList()
protected abstract Set<String>	getWordList(InputStream stream)
void	setExceptionFile(String filename)
void	setMaximumWordLength(int len) Words longer than this will throw an IllegalArgumentException to avoid extremely long processing times.
void	setMinimumWordLength(int len)
void	setStrictMode(boolean strictMode) When set to true, words will only be split if all parts are words.
List<String>	splitWord(String word)
List<String>	splitWord(String word, boolean collectSubwords)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

words

protected Set<String> words

Constructor Detail

AbstractWordSplitter

public AbstractWordSplitter(boolean hideInterfixCharacters)
                     throws IOException

Create a word splitter that uses the embedded dictionary.

Parameters:

hideInterfixCharacters - whether the word parts returned by splitWord(String) still contain the connecting character (a.k.a. interfix)

Throws:

IOException

AbstractWordSplitter

public AbstractWordSplitter(boolean hideInterfixCharacters,
                            InputStream plainTextDict)
                     throws IOException

Parameters:

hideInterfixCharacters - whether the word parts returned by splitWord(String) still contain the connecting character (a.k.a. interfix)

plainTextDict - a stream of a text file with one word per line, to be used instead of the embedded dictionary, must be in UTF-8 format

Throws:

IOException

AbstractWordSplitter

public AbstractWordSplitter(boolean hideInterfixCharacters,
                            File plainTextDict)
                     throws IOException

Parameters:

hideInterfixCharacters - whether the word parts returned by splitWord(String) still contain the connecting character (a.k.a. interfix)

plainTextDict - a stream of a text file with one word per line, to be used instead of the embedded dictionary, must be in UTF-8 format

Throws:

IOException

AbstractWordSplitter

public AbstractWordSplitter(boolean hideInterfixCharacters,
                            Set<String> words)
                     throws IOException

Parameters:

hideInterfixCharacters - whether the word parts returned by splitWord(String) still contain the connecting character (a.k.a. interfix)

words - the compound part words

Throws:

IOException

Since:

4.1

Method Detail

getWordList

protected abstract Set<String> getWordList(InputStream stream)
                                    throws IOException

Throws:

IOException

getWordList

protected abstract Set<String> getWordList()
                                    throws IOException

Throws:

IOException

getDisambiguator

protected abstract de.danielnaber.jwordsplitter.GermanInterfixDisambiguator getDisambiguator()

getDefaultMinimumWordLength

protected abstract int getDefaultMinimumWordLength()

getInterfixCharacters

protected abstract Collection<String> getInterfixCharacters()

Interfix elements in lowercase, e.g. at least "s" for German.

setMinimumWordLength

public void setMinimumWordLength(int len)

setMaximumWordLength

public void setMaximumWordLength(int len)

Words longer than this will throw an IllegalArgumentException to avoid extremely long processing times. The default is 70.

Since:

4.2

setExceptionFile

public void setExceptionFile(String filename)
                      throws IOException

Parameters:

filename - UTF-8 encoded file with exceptions in the classpath, one exception per line, using pipe as delimiter. Example: Pilot|sendung

Throws:

IOException

addException

public void addException(String completeWord,
                         List<String> wordParts)

Parameters:

completeWord - the word for which an exception is to be defined (will be considered case-insensitive)

wordParts - the parts in which the word is to be split (use a list with a single element if the word should not be split)

setStrictMode

public void setStrictMode(boolean strictMode)

When set to true, words will only be split if all parts are words. Otherwise the splitting result might contain parts that are not words.

getAllSplits

public List<List<String>> getAllSplits(String word)

Experimental: Split a word with unknown parts, typically because one part has a typo. This could be used to split three-part compounds where one part has a typo (the caller is then responsible for making useful corrections out of these parts). Results are returned in no specific order.

Since:

4.0

getSubWords

public List<String> getSubWords(String word)

Since:

4.2

splitWord

public List<String> splitWord(String word)

splitWord

public List<String> splitWord(String word,
                              boolean collectSubwords)

Returns:

a list of compound parts, with one element (the input word itself) if the input could not be split; returns an empty list if the input is null

Since:

4.2