ai.tock.nlp.model.service.engine

Interface Tokenizer

All Known Implementing Classes:

NlpTokenizer

@ThreadSafe
public Tokenizer

Method Summary

Modifier and Type	Method and Description
java.lang.String[]	tokenize(ai.tock.nlp.model.TokenizerContext context, java.lang.String text)

Method Detail

tokenize

@NotNull
java.lang.String[] tokenize(@NotNull
                                     ai.tock.nlp.model.TokenizerContext context,
                                     @NotNull
                                     java.lang.String text)