org.eclipse.xtext.parser.antlr

Class XtextTokenStream

java.lang.Object

org.antlr.runtime.CommonTokenStream

org.eclipse.xtext.parser.antlr.XtextTokenStream

All Implemented Interfaces:

org.antlr.runtime.IntStream, org.antlr.runtime.TokenStream

public class XtextTokenStream
extends org.antlr.runtime.CommonTokenStream

A token stream that is aware of the current lookahead.

Author:

Jan Köhnlein - Initial contribution and API, Sebastian Zarnekow - Support for dynamic hidden tokens, reworked lookahead algorithm

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	XtextTokenStream.HiddenTokens

Field Summary

Fields inherited from class org.antlr.runtime.CommonTokenStream

channel, channelOverrideMap, discardOffChannelTokens, discardSet, lastMarker, p, tokens, tokenSource

Constructor Summary

Constructors

Constructor and Description
XtextTokenStream()
XtextTokenStream(org.antlr.runtime.TokenSource tokenSource, int channel)
XtextTokenStream(org.antlr.runtime.TokenSource tokenSource, ITokenDefProvider tokenDefProvider)

Method Summary

Methods

Modifier and Type	Method and Description
void	consume()
protected void	doSetHiddenTokens(String... lexerRules)
int	getCurrentLookAhead()
protected int	getFirstMarker()
String	getLexerErrorMessage(org.antlr.runtime.Token invalidToken)
void	initCurrentLookAhead(int currentLookAhead)
int	LA(int i)
org.antlr.runtime.Token	LT(int k) Same as CommonTokenStream.LT(int) except that we skip hidden tokens even for k == 1.
int	mark()
void	seek(int index)
XtextTokenStream.HiddenTokens	setHiddenTokens(String... lexerRules)
void	setInitialHiddenTokens(String... lexerRules)
protected void	skipHiddenTokens()
String	toString(int start, int stop)

Methods inherited from class org.antlr.runtime.CommonTokenStream

discardOffChannelTokens, discardTokenType, fillBuffer, get, getSourceName, getTokens, getTokens, getTokens, getTokens, getTokens, getTokenSource, index, LB, release, reset, rewind, rewind, setTokenSource, setTokenTypeChannel, size, skipOffTokenChannels, skipOffTokenChannelsReverse, toString, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

XtextTokenStream

public XtextTokenStream()

XtextTokenStream

public XtextTokenStream(org.antlr.runtime.TokenSource tokenSource,
                int channel)

XtextTokenStream

public XtextTokenStream(org.antlr.runtime.TokenSource tokenSource,
                ITokenDefProvider tokenDefProvider)

Method Detail

toString

public String toString(int start,
              int stop)

Specified by:

toString in interface org.antlr.runtime.TokenStream

Overrides:

toString in class org.antlr.runtime.CommonTokenStream

setHiddenTokens

public XtextTokenStream.HiddenTokens setHiddenTokens(String... lexerRules)

setInitialHiddenTokens

public void setInitialHiddenTokens(String... lexerRules)

doSetHiddenTokens

protected void doSetHiddenTokens(String... lexerRules)

skipHiddenTokens

protected void skipHiddenTokens()

LA

public int LA(int i)

Specified by:

LA in interface org.antlr.runtime.IntStream

Overrides:

LA in class org.antlr.runtime.CommonTokenStream

consume

public void consume()

Specified by:

consume in interface org.antlr.runtime.IntStream

Overrides:

consume in class org.antlr.runtime.CommonTokenStream

getFirstMarker

protected int getFirstMarker()

Since:

2.1

mark

public int mark()

Specified by:

mark in interface org.antlr.runtime.IntStream

Overrides:

mark in class org.antlr.runtime.CommonTokenStream

seek

public void seek(int index)

Specified by:

seek in interface org.antlr.runtime.IntStream

Overrides:

seek in class org.antlr.runtime.CommonTokenStream

LT

public org.antlr.runtime.Token LT(int k)

Same as CommonTokenStream.LT(int) except that we skip hidden tokens even for k == 1.

Specified by:

LT in interface org.antlr.runtime.TokenStream

Overrides:

LT in class org.antlr.runtime.CommonTokenStream

getCurrentLookAhead

public int getCurrentLookAhead()

initCurrentLookAhead

public void initCurrentLookAhead(int currentLookAhead)

getLexerErrorMessage

public String getLexerErrorMessage(org.antlr.runtime.Token invalidToken)