public class CommonTokenStream extends BufferedTokenStream
BufferedTokenStream with functionality to filter
token streams to tokens on a particular channel (tokens where
Token.getChannel() returns a particular value).
This token stream provides access to all tokens by index or when calling
methods like BufferedTokenStream.getText(). The channel filtering is only used for code
accessing tokens via the lookahead methods BufferedTokenStream.LA(int), CommonTokenStream.LT(int), and
CommonTokenStream.LB(int).
By default, tokens are placed on the default channel
(Token.DEFAULT_CHANNEL), but may be reassigned by using the
->channel(HIDDEN) lexer command, or by using an embedded action to
call Lexer.setChannel(int).
Note: lexer rules which use the ->skip lexer command or call
Lexer.skip() do not produce tokens at all, so input text matched by
such a rule will not be available as part of the token stream, regardless of
channel.
| Modifier and Type | Field and Description |
|---|---|
protected int |
channel
Specifies the channel to use for filtering tokens.
|
fetchedEOF, p, tokens, tokenSourceEOF, UNKNOWN_SOURCE_NAME| Constructor and Description |
|---|
CommonTokenStream(TokenSource tokenSource)
Constructs a new
CommonTokenStream using the specified token
source and the default token channel (Token.DEFAULT_CHANNEL). |
CommonTokenStream(TokenSource tokenSource,
int channel)
Constructs a new
CommonTokenStream using the specified token
source and filtering tokens to the specified channel. |
| Modifier and Type | Method and Description |
|---|---|
protected int |
adjustSeekIndex(int i)
Allowed derived classes to modify the behavior of operations which change
the current stream position by adjusting the target token index of a seek
operation.
|
int |
getNumberOfOnChannelTokens()
Count EOF just once.
|
protected Token |
LB(int k) |
Token |
LT(int k)
|
consume, fetch, fill, filterForChannel, get, get, getHiddenTokensToLeft, getHiddenTokensToLeft, getHiddenTokensToRight, getHiddenTokensToRight, getSourceName, getText, getText, getText, getText, getTokens, getTokens, getTokens, getTokens, getTokenSource, index, LA, lazyInit, mark, nextTokenOnChannel, previousTokenOnChannel, release, seek, setTokenSource, setup, size, syncprotected int channel
The default value is Token.DEFAULT_CHANNEL, which matches the
default channel assigned to tokens created by the lexer.
public CommonTokenStream(TokenSource tokenSource)
CommonTokenStream using the specified token
source and the default token channel (Token.DEFAULT_CHANNEL).tokenSource - The token source.public CommonTokenStream(TokenSource tokenSource, int channel)
CommonTokenStream using the specified token
source and filtering tokens to the specified channel. Only tokens whose
Token.getChannel() matches channel or have the
Token.getType() equal to Token.EOF will be returned by the
token stream lookahead methods.tokenSource - The token source.channel - The channel to use for filtering tokens.protected int adjustSeekIndex(int i)
BufferedTokenStreami. If an
exception is thrown in this method, the current stream index should not be
changed.
For example, CommonTokenStream overrides this method to ensure that
the seek target is always an on-channel token.
adjustSeekIndex in class BufferedTokenStreami - The target token index.protected Token LB(int k)
LB in class BufferedTokenStreampublic Token LT(int k)
TokenStreamToken instance associated with the value returned by
LA(k). This method has the same pre- and post-conditions as
IntStream.LA(int). In addition, when the preconditions of this method
are met, the return value is non-null and the value of
LT(k).getType()==LA(k).LT in interface TokenStreamLT in class BufferedTokenStreamIntStream.LA(int)public int getNumberOfOnChannelTokens()