org.deeplearning4j.nn.modelimport.keras.preprocessing.text

Class KerasTokenizer

java.lang.Object

org.deeplearning4j.nn.modelimport.keras.preprocessing.text.KerasTokenizer

public class KerasTokenizer
extends Object

Constructor Summary

Constructors

Constructor and Description
KerasTokenizer() Default Keras tokenizer constructor
KerasTokenizer(Integer numWords) Tokenizer constructor with only numWords specified
KerasTokenizer(Integer numWords, String filters, boolean lower, String split, boolean charLevel, String outOfVocabularyToken) Create a Keras Tokenizer instance with full set of properties.

Method Summary

Modifier and Type	Method and Description
void	fitOnSequences(Integer[][] sequences) Fit this tokenizer on a corpus of word indices
void	fitOnTexts(String[] texts) Fit this tokenizer on a corpus of texts.
static KerasTokenizer	fromJson(String jsonFileName) Import Keras Tokenizer from JSON file created with `tokenizer.to_json()` in Python.
INDArray	sequencesToMatrix(Integer[][] sequences, TokenizerMode mode) Turns an array of index sequences into an ND4J matrix of shape (number of texts, number of words in vocabulary)
String[]	sequencesToTexts(Integer[][] sequences) Turns index sequences back into texts
INDArray	textsToMatrix(String[] texts, TokenizerMode mode) Turns an array of texts into an ND4J matrix of shape (number of texts, number of words in vocabulary)
Integer[][]	textsToSequences(String[] texts) Transforms a bunch of texts into their index representations.
static String[]	textToWordSequence(String text, String filters, boolean lower, String split) Turns a String text into a sequence of tokens.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

KerasTokenizer

public KerasTokenizer(Integer numWords,
                      String filters,
                      boolean lower,
                      String split,
                      boolean charLevel,
                      String outOfVocabularyToken)

Create a Keras Tokenizer instance with full set of properties.

Parameters:

numWords - The maximum vocabulary size, can be null

filters - Characters to filter

lower - whether to lowercase input or not

split - by which string to split words (usually single space)

charLevel - whether to operate on character- or word-level

outOfVocabularyToken - replace items outside the vocabulary by this token

KerasTokenizer

public KerasTokenizer(Integer numWords)

Tokenizer constructor with only numWords specified

Parameters:

numWords - The maximum vocabulary size, can be null

KerasTokenizer

public KerasTokenizer()

Default Keras tokenizer constructor

Method Detail

fromJson

public static KerasTokenizer fromJson(String jsonFileName)
                               throws IOException,
                                      InvalidKerasConfigurationException

Import Keras Tokenizer from JSON file created with `tokenizer.to_json()` in Python.

Parameters:

jsonFileName - Full path of the JSON file to load

Returns:

Keras Tokenizer instance loaded from JSON

Throws:

IOException - I/O exception

InvalidKerasConfigurationException - Invalid Keras configuration

textToWordSequence

public static String[] textToWordSequence(String text,
                                          String filters,
                                          boolean lower,
                                          String split)

Turns a String text into a sequence of tokens.

Parameters:

text - input text

filters - characters to filter

lower - whether to lowercase input or not

split - by which string to split words (usually single space)

Returns:

Sequence of tokens as String array

fitOnTexts

public void fitOnTexts(String[] texts)

Fit this tokenizer on a corpus of texts.

Parameters:

texts - array of strings to fit tokenizer on.

fitOnSequences

public void fitOnSequences(Integer[][] sequences)

Fit this tokenizer on a corpus of word indices

Parameters:

sequences - array of indices derived from a text.

textsToSequences

public Integer[][] textsToSequences(String[] texts)

Transforms a bunch of texts into their index representations.

Parameters:

texts - input texts

Returns:

array of indices of the texts

sequencesToTexts

public String[] sequencesToTexts(Integer[][] sequences)

Turns index sequences back into texts

Parameters:

sequences - index sequences

Returns:

text reconstructed from sequences

textsToMatrix

public INDArray textsToMatrix(String[] texts,
                              TokenizerMode mode)

Turns an array of texts into an ND4J matrix of shape (number of texts, number of words in vocabulary)

Parameters:

texts - input texts

mode - TokenizerMode that controls how to vectorize data

Returns:

resulting matrix representation

sequencesToMatrix

public INDArray sequencesToMatrix(Integer[][] sequences,
                                  TokenizerMode mode)

Turns an array of index sequences into an ND4J matrix of shape (number of texts, number of words in vocabulary)

Parameters:

sequences - input sequences

mode - TokenizerMode that controls how to vectorize data

Returns:

resulting matrix representatio