public class CommonToken extends Object implements WritableToken, Serializable
| Modifier and Type | Field and Description |
|---|---|
protected int |
channel
This is the backing field for
CommonToken.getChannel() and
CommonToken.setChannel(int). |
protected int |
charPositionInLine
This is the backing field for
CommonToken.getCharPositionInLine() and
CommonToken.setCharPositionInLine(int). |
protected static Pair<TokenSource,CharStream> |
EMPTY_SOURCE
An empty
Pair which is used as the default value of
CommonToken.source for tokens that do not have a source. |
protected int |
index
This is the backing field for
CommonToken.getTokenIndex() and
CommonToken.setTokenIndex(int). |
protected int |
line
This is the backing field for
CommonToken.getLine() and CommonToken.setLine(int). |
protected Pair<TokenSource,CharStream> |
source
This is the backing field for
CommonToken.getTokenSource() and
CommonToken.getInputStream(). |
protected int |
start
This is the backing field for
CommonToken.getStartIndex() and
CommonToken.setStartIndex(int). |
protected int |
stop
This is the backing field for
CommonToken.getStopIndex() and
CommonToken.setStopIndex(int). |
protected String |
text
This is the backing field for
CommonToken.getText() when the token text is
explicitly set in the constructor or via CommonToken.setText(java.lang.String). |
protected int |
type
This is the backing field for
CommonToken.getType() and CommonToken.setType(int). |
DEFAULT_CHANNEL, EOF, EPSILON, HIDDEN_CHANNEL, INVALID_TYPE, MIN_USER_CHANNEL_VALUE, MIN_USER_TOKEN_TYPE| Constructor and Description |
|---|
CommonToken(int type)
Constructs a new
CommonToken with the specified token type. |
CommonToken(int type,
String text)
Constructs a new
CommonToken with the specified token type and
text. |
CommonToken(Pair<TokenSource,CharStream> source,
int type,
int channel,
int start,
int stop) |
CommonToken(Token oldToken)
Constructs a new
CommonToken as a copy of another Token. |
| Modifier and Type | Method and Description |
|---|---|
int |
getChannel()
Return the channel this token.
|
int |
getCharPositionInLine()
The index of the first character of this token relative to the
beginning of the line at which it occurs, 0..n-1
|
CharStream |
getInputStream()
Gets the
CharStream from which this token was derived. |
int |
getLine()
The line number on which the 1st character of this token was matched,
line=1..n
|
int |
getStartIndex()
The starting character index of the token
This method is optional; return -1 if not implemented.
|
int |
getStopIndex()
The last character index of the token.
|
String |
getText()
Get the text of the token.
|
int |
getTokenIndex()
An index from 0..n-1 of the token object in the input stream.
|
TokenSource |
getTokenSource()
Gets the
TokenSource which created this token. |
int |
getType()
Get the token type of the token
|
void |
setChannel(int channel) |
void |
setCharPositionInLine(int charPositionInLine) |
void |
setLine(int line) |
void |
setStartIndex(int start) |
void |
setStopIndex(int stop) |
void |
setText(String text)
Explicitly set the text for this token.
|
void |
setTokenIndex(int index) |
void |
setType(int type) |
String |
toString() |
String |
toString(Recognizer<?,?> r) |
protected static final Pair<TokenSource,CharStream> EMPTY_SOURCE
Pair which is used as the default value of
CommonToken.source for tokens that do not have a source.protected int type
CommonToken.getType() and CommonToken.setType(int).protected int line
CommonToken.getLine() and CommonToken.setLine(int).protected int charPositionInLine
CommonToken.getCharPositionInLine() and
CommonToken.setCharPositionInLine(int).protected int channel
CommonToken.getChannel() and
CommonToken.setChannel(int).protected Pair<TokenSource,CharStream> source
CommonToken.getTokenSource() and
CommonToken.getInputStream().
These properties share a field to reduce the memory footprint of
CommonToken. Tokens created by a CommonTokenFactory from
the same source and input stream share a reference to the same
Pair containing these values.
protected String text
CommonToken.getText() when the token text is
explicitly set in the constructor or via CommonToken.setText(java.lang.String).CommonToken.getText()protected int index
CommonToken.getTokenIndex() and
CommonToken.setTokenIndex(int).protected int start
CommonToken.getStartIndex() and
CommonToken.setStartIndex(int).protected int stop
CommonToken.getStopIndex() and
CommonToken.setStopIndex(int).public CommonToken(int type)
CommonToken with the specified token type.type - The token type.public CommonToken(Pair<TokenSource,CharStream> source, int type, int channel, int start, int stop)
public CommonToken(int type,
String text)
CommonToken with the specified token type and
text.type - The token type.text - The text of the token.public CommonToken(Token oldToken)
CommonToken as a copy of another Token.
If oldToken is also a CommonToken instance, the newly
constructed token will share a reference to the CommonToken.text field and
the Pair stored in CommonToken.source. Otherwise, CommonToken.text will
be assigned the result of calling CommonToken.getText(), and CommonToken.source
will be constructed from the result of Token.getTokenSource() and
Token.getInputStream().
oldToken - The token to copy.public int getType()
Tokenpublic void setLine(int line)
setLine in interface WritableTokenpublic void setText(String text)
null, then CommonToken.getText() will return this value rather than
extracting the text from the input.setText in interface WritableTokentext - The explicit text of the token, or null if the text
should be obtained from the input along with the start and stop indexes
of the token.public int getLine()
Tokenpublic int getCharPositionInLine()
TokengetCharPositionInLine in interface Tokenpublic void setCharPositionInLine(int charPositionInLine)
setCharPositionInLine in interface WritableTokenpublic int getChannel()
TokengetChannel in interface Tokenpublic void setChannel(int channel)
setChannel in interface WritableTokenpublic void setType(int type)
setType in interface WritableTokenpublic int getStartIndex()
TokengetStartIndex in interface Tokenpublic void setStartIndex(int start)
public int getStopIndex()
TokengetStopIndex in interface Tokenpublic void setStopIndex(int stop)
public int getTokenIndex()
TokengetTokenIndex in interface Tokenpublic void setTokenIndex(int index)
setTokenIndex in interface WritableTokenpublic TokenSource getTokenSource()
TokenTokenSource which created this token.getTokenSource in interface Tokenpublic CharStream getInputStream()
TokenCharStream from which this token was derived.getInputStream in interface Tokenpublic String toString(Recognizer<?,?> r)