Class CommonTokenStream

java.lang.Object
org.graalvm.shadowed.org.antlr.v4.runtime.BufferedTokenStream
org.graalvm.shadowed.org.antlr.v4.runtime.CommonTokenStream
All Implemented Interfaces:
IntStream, TokenStream
public class CommonTokenStream extends BufferedTokenStream
This class extends BufferedTokenStream with functionality to filter token streams to tokens on a particular channel (tokens where Token.getChannel() returns a particular value).

This token stream provides access to all tokens by index or when calling methods like BufferedTokenStream.getText(). The channel filtering is only used for code accessing tokens via the lookahead methods BufferedTokenStream.LA(int), LT(int), and LB(int).

By default, tokens are placed on the default channel (Token.DEFAULT_CHANNEL), but may be reassigned by using the ->channel(HIDDEN) lexer command, or by using an embedded action to call Lexer.setChannel(int).

Note: lexer rules which use the ->skip lexer command or call Lexer.skip() do not produce tokens at all, so input text matched by such a rule will not be available as part of the token stream, regardless of channel.

we

  • Field Details

    • channel

      protected int channel
      Specifies the channel to use for filtering tokens.

      The default value is Token.DEFAULT_CHANNEL, which matches the default channel assigned to tokens created by the lexer.

  • Constructor Details

    • CommonTokenStream

      public CommonTokenStream(TokenSource tokenSource)
      Constructs a new CommonTokenStream using the specified token source and the default token channel (Token.DEFAULT_CHANNEL).
      Parameters:
      tokenSource - The token source.

    • CommonTokenStream

      public CommonTokenStream(TokenSource tokenSource, int channel)
      Constructs a new CommonTokenStream using the specified token source and filtering tokens to the specified channel. Only tokens whose Token.getChannel() matches channel or have the Token.getType() equal to Token.EOF will be returned by the token stream lookahead methods.
      Parameters:
      tokenSource - The token source.
      channel - The channel to use for filtering tokens.

  • Method Details

    • adjustSeekIndex

      protected int adjustSeekIndex(int i)
      Description copied from class: BufferedTokenStream
      Allowed derived classes to modify the behavior of operations which change the current stream position by adjusting the target token index of a seek operation. The default implementation simply returns i. If an exception is thrown in this method, the current stream index should not be changed.

      For example, CommonTokenStream overrides this method to ensure that the seek target is always an on-channel token.

      Overrides:
      adjustSeekIndex in class BufferedTokenStream
      Parameters:
      i - The target token index.
      Returns:
      The adjusted target token index.

    • LB

      protected Token LB(int k)
      Overrides:
      LB in class BufferedTokenStream

    • LT

      public Token LT(int k)
      Description copied from interface: TokenStream
      Get the Token instance associated with the value returned by LA(k). This method has the same pre- and post-conditions as IntStream.LA(int). In addition, when the preconditions of this method are met, the return value is non-null and the value of LT(k).getType()==LA(k).
      Specified by:
      LT in interface TokenStream
      Overrides:
      LT in class BufferedTokenStream
      See Also:

    • getNumberOfOnChannelTokens

      public int getNumberOfOnChannelTokens()
      Count EOF just once.