org.eclipse.xtext.parser.antlr

Class AbstractIndentationTokenSource

java.lang.Object

org.eclipse.xtext.parser.antlr.AbstractSplittingTokenSource

org.eclipse.xtext.parser.antlr.AbstractIndentationTokenSource

All Implemented Interfaces:

org.antlr.runtime.TokenSource

public abstract class AbstractIndentationTokenSource
extends AbstractSplittingTokenSource

Abstract implementation of a token source, that splits according to indentation levels after newlines. Blank lines are ignored. At the end of file token, all pending indentations will be handled and dedentation tokens will be issued accordingly if shouldEmitPendingEndTokens() returns true.

Since:

2.8

Field Summary

Fields

Modifier and Type	Field and Description
protected int	currentEnd
protected int	currentIndentation
protected int	currentOffset
protected java.util.Stack<java.lang.Integer>	indentationStack
protected int	nextOffset

Fields inherited from class org.eclipse.xtext.parser.antlr.AbstractSplittingTokenSource

LEXER_RULE_PREFIX

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	AbstractIndentationTokenSource(org.antlr.runtime.TokenSource delegate)

Method Summary

Modifier and Type	Method and Description
protected int	computeIndentation(java.lang.String text)
protected int	computeIndentationRelevantCharCount(java.lang.String text)
protected org.antlr.runtime.Token	createBeginToken(int offset)
protected org.antlr.runtime.Token	createEndToken(int offset)
protected void	decreaseIndentation(int indentation, ITokenAcceptor result)
protected void	doSplitEofToken(org.antlr.runtime.Token token, ITokenAcceptor result)
protected void	doSplitToken(org.antlr.runtime.Token token, ITokenAcceptor result)
protected void	doSplitTokenImpl(org.antlr.runtime.Token token, ITokenAcceptor result) The token was previously determined as potentially to-be-splitted thus we emit additional indentation or dedenting tokens.
protected abstract int	getBeginTokenType()
protected abstract int	getEndTokenType()
protected char	getTabWidth()
protected void	handleRemainingText(org.antlr.runtime.Token token, java.lang.String text, int indentation, ITokenAcceptor result)
org.antlr.runtime.Token	nextToken()
protected boolean	shouldEmitPendingEndTokens()
protected boolean	shouldSplitToken(org.antlr.runtime.Token token)
protected abstract boolean	shouldSplitTokenImpl(org.antlr.runtime.Token token) Return true if the given token is a whitespace token that should be splitted.
protected void	splitWithText(org.antlr.runtime.Token token, java.lang.String text, ITokenAcceptor result)

Methods inherited from class org.eclipse.xtext.parser.antlr.AbstractSplittingTokenSource

createToken, getDelegate, getLexerRuleName, getRuleName, getSourceName, getTokenType, initializeTokenDefsFrom, setDelegate, splitToken

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

indentationStack

protected java.util.Stack<java.lang.Integer> indentationStack

currentOffset

protected int currentOffset

currentIndentation

protected int currentIndentation

currentEnd

protected int currentEnd

nextOffset

protected int nextOffset

Constructor Detail

AbstractIndentationTokenSource

protected AbstractIndentationTokenSource(org.antlr.runtime.TokenSource delegate)

Method Detail

shouldSplitToken

protected boolean shouldSplitToken(org.antlr.runtime.Token token)

Specified by:

shouldSplitToken in class AbstractSplittingTokenSource

shouldSplitTokenImpl

protected abstract boolean shouldSplitTokenImpl(org.antlr.runtime.Token token)

Return true if the given token is a whitespace token that should be splitted. Usually the token type will be compared to one of the int constants from the parser.

doSplitEofToken

protected void doSplitEofToken(org.antlr.runtime.Token token,
                               ITokenAcceptor result)

Since:

2.13

doSplitToken

protected void doSplitToken(org.antlr.runtime.Token token,
                            ITokenAcceptor result)

Specified by:

doSplitToken in class AbstractSplittingTokenSource

shouldEmitPendingEndTokens

protected boolean shouldEmitPendingEndTokens()

nextToken

public org.antlr.runtime.Token nextToken()

Specified by:

nextToken in interface org.antlr.runtime.TokenSource

Overrides:

nextToken in class AbstractSplittingTokenSource

doSplitTokenImpl

protected void doSplitTokenImpl(org.antlr.runtime.Token token,
                                ITokenAcceptor result)

The token was previously determined as potentially to-be-splitted thus we emit additional indentation or dedenting tokens.

handleRemainingText

protected void handleRemainingText(org.antlr.runtime.Token token,
                                   java.lang.String text,
                                   int indentation,
                                   ITokenAcceptor result)

Since:

2.13

decreaseIndentation

protected void decreaseIndentation(int indentation,
                                   ITokenAcceptor result)

splitWithText

protected void splitWithText(org.antlr.runtime.Token token,
                             java.lang.String text,
                             ITokenAcceptor result)

createEndToken

protected org.antlr.runtime.Token createEndToken(int offset)

getEndTokenType

protected abstract int getEndTokenType()

getBeginTokenType

protected abstract int getBeginTokenType()

createBeginToken

protected org.antlr.runtime.Token createBeginToken(int offset)

computeIndentationRelevantCharCount

protected int computeIndentationRelevantCharCount(java.lang.String text)

computeIndentation

protected int computeIndentation(java.lang.String text)

getTabWidth

protected char getTabWidth()