ai.djl.modality.nlp.preprocess

Class PunctuationSeparator

java.lang.Object

ai.djl.modality.nlp.preprocess.PunctuationSeparator

All Implemented Interfaces:

TextProcessor

public class PunctuationSeparator
extends java.lang.Object
implements TextProcessor

PunctuationSeparator converts every character of the input tokens to it's respective lower case character.

Constructor Summary

Constructors

Constructor and Description
PunctuationSeparator() Creates a TextProcessor that separates the given punctuations into distinct tokens in the text.
PunctuationSeparator(java.lang.String punctuations) Creates a TextProcessor that separates the given punctuations into distinct tokens in the text.

Method Summary

Modifier and Type	Method and Description
java.util.List<java.lang.String>	preprocess(java.util.List<java.lang.String> tokens) Applies the preprocessing defined to the given input tokens.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PunctuationSeparator

public PunctuationSeparator(java.lang.String punctuations)

Creates a TextProcessor that separates the given punctuations into distinct tokens in the text.

Parameters:

punctuations - the punctuations to be separated

PunctuationSeparator

public PunctuationSeparator()

Creates a TextProcessor that separates the given punctuations into distinct tokens in the text.

Method Detail

preprocess

public java.util.List<java.lang.String> preprocess(java.util.List<java.lang.String> tokens)

Applies the preprocessing defined to the given input tokens.

Specified by:

preprocess in interface TextProcessor

Parameters:

tokens - the tokens created after the input text is tokenized

Returns:

the preprocessed tokens