Package org.apache.lucene.analysis.standard

Interface StandardTokenizerInterface

All Known Implementing Classes:
StandardTokenizerImpl, StandardTokenizerImpl31, StandardTokenizerImpl34, StandardTokenizerImpl40, UAX29URLEmailTokenizerImpl, UAX29URLEmailTokenizerImpl31, UAX29URLEmailTokenizerImpl34, UAX29URLEmailTokenizerImpl36, UAX29URLEmailTokenizerImpl40
public interface StandardTokenizerInterface
Internal interface for supporting versioned grammars.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final int
    YYEOF
    This character denotes the end of file

  • Method Summary

    Modifier and Type
    Method
    Description
    int
    getNextToken()
    Resumes scanning until the next regular expression is matched, the end of input is encountered or an I/O-Error occurs.
    void
    getText(CharTermAttribute t)
    Copies the matched text into the CharTermAttribute
    int
    yychar()
    Returns the current position.
    int
    yylength()
    Returns the length of the matched text region.
    void
    yyreset(Reader reader)
    Resets the scanner to read from a new input stream.

  • Field Details

  • Method Details

    • getText

      void getText(CharTermAttribute t)
      Copies the matched text into the CharTermAttribute

    • yychar

      int yychar()
      Returns the current position.

    • yyreset

      void yyreset(Reader reader)
      Resets the scanner to read from a new input stream. Does not close the old reader. All internal variables are reset, the old input stream cannot be reused (internal buffer is discarded and lost). Lexical state is set to ZZ_INITIAL.
      Parameters:
      reader - the new input stream

    • yylength

      int yylength()
      Returns the length of the matched text region.

    • getNextToken

      int getNextToken() throws IOException
      Resumes scanning until the next regular expression is matched, the end of input is encountered or an I/O-Error occurs.
      Returns:
      the next token, YYEOF on end of stream
      Throws:
      IOException - if any I/O-Error occurs