Interface Token

All Known Subinterfaces:
WritableToken
All Known Implementing Classes:
CommonToken, RuleTagToken, TokenTagToken
public interface Token
A token has properties: text, type, line, character position in the line (so we can ignore tabs), token channel, index, and source from which we obtained this token.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final int
    DEFAULT_CHANNEL
    All tokens go to the parser (unless skip() is called in that rule) on a particular "channel".
    static final int
    EOF
     
    static final int
    EPSILON
    During lookahead operations, this "token" signifies we hit rule end ATN state and did not follow it despite needing to.
    static final int
    HIDDEN_CHANNEL
    Anything on different channel than DEFAULT_CHANNEL is not parsed by parser.
    static final int
    INVALID_TYPE
     
    static final int
    MIN_USER_CHANNEL_VALUE
    This is the minimum constant value which can be assigned to a user-defined token channel.
    static final int
    MIN_USER_TOKEN_TYPE
     

  • Method Summary

    Modifier and Type
    Method
    Description
    int
    getChannel()
    Return the channel this token.
    int
    getCharPositionInLine()
    The index of the first character of this token relative to the beginning of the line at which it occurs, 0..n-1
    CharStream
    getInputStream()
    Gets the CharStream from which this token was derived.
    int
    getLine()
    The line number on which the 1st character of this token was matched, line=1..n
    int
    getStartIndex()
    The starting character index of the token This method is optional; return -1 if not implemented.
    int
    getStopIndex()
    The last character index of the token.
    String
    getText()
    Get the text of the token.
    int
    getTokenIndex()
    An index from 0..n-1 of the token object in the input stream.
    TokenSource
    getTokenSource()
    Gets the TokenSource which created this token.
    int
    getType()
    Get the token type of the token

  • Field Details

    • INVALID_TYPE

      static final int INVALID_TYPE
      See Also:

    • EPSILON

      static final int EPSILON
      During lookahead operations, this "token" signifies we hit rule end ATN state and did not follow it despite needing to.
      See Also:

    • MIN_USER_TOKEN_TYPE

      static final int MIN_USER_TOKEN_TYPE
      See Also:

    • EOF

      static final int EOF
      See Also:

    • DEFAULT_CHANNEL

      static final int DEFAULT_CHANNEL
      All tokens go to the parser (unless skip() is called in that rule) on a particular "channel". The parser tunes to a particular channel so that whitespace etc... can go to the parser on a "hidden" channel.
      See Also:

    • HIDDEN_CHANNEL

      static final int HIDDEN_CHANNEL
      Anything on different channel than DEFAULT_CHANNEL is not parsed by parser.
      See Also:

    • MIN_USER_CHANNEL_VALUE

      static final int MIN_USER_CHANNEL_VALUE
      This is the minimum constant value which can be assigned to a user-defined token channel.

      The non-negative numbers less than MIN_USER_CHANNEL_VALUE are assigned to the predefined channels DEFAULT_CHANNEL and HIDDEN_CHANNEL.

      See Also:

  • Method Details

    • getText

      String getText()
      Get the text of the token.

    • getType

      int getType()
      Get the token type of the token

    • getLine

      int getLine()
      The line number on which the 1st character of this token was matched, line=1..n

    • getCharPositionInLine

      int getCharPositionInLine()
      The index of the first character of this token relative to the beginning of the line at which it occurs, 0..n-1

    • getChannel

      int getChannel()
      Return the channel this token. Each token can arrive at the parser on a different channel, but the parser only "tunes" to a single channel. The parser ignores everything not on DEFAULT_CHANNEL.

    • getTokenIndex

      int getTokenIndex()
      An index from 0..n-1 of the token object in the input stream. This must be valid in order to print token streams and use TokenRewriteStream. Return -1 to indicate that this token was conjured up since it doesn't have a valid index.

    • getStartIndex

      int getStartIndex()
      The starting character index of the token This method is optional; return -1 if not implemented.

    • getStopIndex

      int getStopIndex()
      The last character index of the token. This method is optional; return -1 if not implemented.

    • getTokenSource

      TokenSource getTokenSource()
      Gets the TokenSource which created this token.

    • getInputStream

      CharStream getInputStream()
      Gets the CharStream from which this token was derived.