Class CommonTokenFactory
java.lang.Object
org.graalvm.shadowed.org.antlr.v4.runtime.CommonTokenFactory
- All Implemented Interfaces:
TokenFactory<CommonToken>
This default implementation of
TokenFactory creates
CommonToken objects.-
Field Summary
FieldsModifier and TypeFieldDescriptionprotected final booleanIndicates whetherCommonToken.setText(java.lang.String)should be called after constructing tokens to explicitly set the text.static final TokenFactory<CommonToken> The defaultCommonTokenFactoryinstance. -
Constructor Summary
ConstructorsConstructorDescriptionCommonTokenFactory(boolean copyText) Constructs aCommonTokenFactorywith the specified value forcopyText. -
Method Summary
Modifier and TypeMethodDescriptionGenerically usefulcreate(Pair<TokenSource, CharStream> source, int type, String text, int channel, int start, int stop, int line, int charPositionInLine) This is the method used to create tokens in the lexer and in the error handling strategy.
-
Field Details
-
DEFAULT
The defaultCommonTokenFactoryinstance.This token factory does not explicitly copy token text when constructing tokens.
-
copyText
protected final boolean copyTextIndicates whetherCommonToken.setText(java.lang.String)should be called after constructing tokens to explicitly set the text. This is useful for cases where the input stream might not be able to provide arbitrary substrings of text from the input after the lexer creates a token (e.g. the implementation ofCharStream.getText(org.graalvm.shadowed.org.antlr.v4.runtime.misc.Interval)inUnbufferedCharStreamthrows anUnsupportedOperationException). Explicitly setting the token text allowsToken.getText()to be called at any time regardless of the input stream implementation.The default value is
falseto avoid the performance and memory overhead of copying text for every token unless explicitly requested.
-
-
Constructor Details
-
CommonTokenFactory
public CommonTokenFactory(boolean copyText) Constructs aCommonTokenFactorywith the specified value forcopyText.When
copyTextisfalse, theDEFAULTinstance should be used instead of constructing a new instance.- Parameters:
copyText- The value forcopyText.
-
CommonTokenFactory
public CommonTokenFactory()Constructs aCommonTokenFactorywithcopyTextset tofalse.The
DEFAULTinstance should be used instead of calling this directly.
-
-
Method Details
-
create
public CommonToken create(Pair<TokenSource, CharStream> source, int type, String text, int channel, int start, int stop, int line, int charPositionInLine) Description copied from interface:TokenFactoryThis is the method used to create tokens in the lexer and in the error handling strategy. If text!=null, than the start and stop positions are wiped to -1 in the text override is set in the CommonToken.- Specified by:
createin interfaceTokenFactory<CommonToken>
-
create
Description copied from interface:TokenFactoryGenerically useful- Specified by:
createin interfaceTokenFactory<CommonToken>
-