Package com.yahoo.language.opennlp

Class OpenNlpTokenizer

java.lang.Object
com.yahoo.language.opennlp.OpenNlpTokenizer
All Implemented Interfaces:
com.yahoo.language.process.Tokenizer
public class OpenNlpTokenizer extends Object implements com.yahoo.language.process.Tokenizer
Tokenizer using OpenNlp
Author:
matskin, bratseth

  • Constructor Summary

    Constructors
    Constructor
    Description
    OpenNlpTokenizer()
     
    OpenNlpTokenizer(com.yahoo.language.process.Normalizer normalizer, com.yahoo.language.process.Transformer transformer)
     
    OpenNlpTokenizer(com.yahoo.language.process.Normalizer normalizer, com.yahoo.language.process.Transformer transformer, com.yahoo.language.process.SpecialTokenRegistry specialTokenRegistry)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    Iterable<com.yahoo.language.process.Token>
    tokenize(String input, com.yahoo.language.Language language, com.yahoo.language.process.StemMode stemMode, boolean removeAccents)
     

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • OpenNlpTokenizer

      public OpenNlpTokenizer()

    • OpenNlpTokenizer

      public OpenNlpTokenizer(com.yahoo.language.process.Normalizer normalizer, com.yahoo.language.process.Transformer transformer)

    • OpenNlpTokenizer

      public OpenNlpTokenizer(com.yahoo.language.process.Normalizer normalizer, com.yahoo.language.process.Transformer transformer, com.yahoo.language.process.SpecialTokenRegistry specialTokenRegistry)

  • Method Details

    • tokenize

      public Iterable<com.yahoo.language.process.Token> tokenize(String input, com.yahoo.language.Language language, com.yahoo.language.process.StemMode stemMode, boolean removeAccents)
      Specified by:
      tokenize in interface com.yahoo.language.process.Tokenizer