org.teatrove.tea.compiler
Class Token

java.lang.Object
  org.teatrove.tea.compiler.Token

All Implemented Interfaces:

Serializable

public class Token
extends Object
implements Serializable

A Token represents the smallest whole element of a source file. Tokens are produced by a Scanner.

Author:

Brian S O'Neill

See Also:

Serialized Form

Field Summary

static int	AND Token ID for the and keyword: 'and'
static int	AS Token ID for the as keyword: 'as'
static int	ASSIGN Token ID for the assignment operator: '='
static int	BREAK Token ID for the break keyword: 'break'
static int	CALL Token ID for the call keyword: 'call'
static int	CLASS Token ID for the class keyword: 'class'
static int	COLON Token ID for colon: ':'
static int	COMMA Token ID for the comma: ','
static int	COMMENT Token ID for a single-line or multi-line comment.
static int	CONCAT Token ID for the concatenation operator: '&'
static int	CONTINUE Token ID for the continue keyword: 'continue'
static int	DEFINE Token ID for the define keyword: 'define'
static int	DIV Token ID for the division operator: '/'
static int	DOT Token ID for the dot: '.'
static int	DOTDOT Token ID for the double dot: '..'
static int	DOUBLE_HASH Token ID for the double hash: '##'
static int	ELLIPSIS Token ID for the ellipsis: '...'
static int	ELSE Token ID for the else keyword: 'else'
static int	ENTER_CODE Token ID for the start of a code region.
static int	ENTER_TEXT Token ID for the start of a text region.
static int	EOF Token ID for the end of file.
static int	EQ Token ID for the equal operator: '=='
static int	EQUAL_GREATER Token ID for equal greater: '=>'
static int	FALSE Token ID for the false literal: 'false'
static int	FOREACH Token ID for the for-each keyword: 'foreach'
static int	GE Token ID for the greater than or equal operator: '>='
static int	GT Token ID for the greater than operator: '>'
static int	HASH Token ID for the hash: '#'
static int	IDENT Token ID for an identifier.
static int	IF Token ID for the if keyword: 'if'
static int	IMPORT Token ID for the import keyword: 'import'
static int	IN Token ID for the in keyword: 'in'
static int	ISA Token ID for the is-a keyword: 'isa'
static int	LAMBDA Token ID for lambda operator: '->'
static int	LBRACE Token ID for the left brace: '{'
static int	LBRACK Token ID for the left bracket: '['
static int	LE Token ID for the less than or equal operator: '<='
static int	LPAREN Token ID for the left parenthesis: '('
static int	LT Token ID for the less than operator: '<'
static int	MINUS Token ID for the minus operator: '-'
static int	MOD Token ID for the modulus operator: '%'
static int	MULT Token ID for the multiplication operator: '*'
static int	NE Token ID for the not equal operator: '!='
static int	NOT Token ID for the not keyword: 'not'
static int	NULL Token ID for the null literal: 'null'
static int	NUMBER Token ID for a number literal.
static int	OR Token ID for the or keyword: 'or'
static int	PLUS Token ID for the plus operator: '+'
static int	QUESTION Token ID for question mark: '?'
static int	RBRACE Token ID for the right brace: '}'
static int	RBRACK Token ID for the right bracket: ']'
static int	REVERSE Token ID for the reverse keyword: 'reverse'
static int	RPAREN Token ID for the right parenthesis: ')'
static int	SEMI Token ID for the semi-colon: ';'
static int	SLASH Token ID for the forward slash: '/'
static int	SPACESHIP Token ID for spaceship (comparator) operator: '<=>'
static int	SPREAD Token ID for spread operator: *.
static int	STRING Token ID for a string literal.
static int	TEMPLATE Token ID for the template keyword: 'template'
static int	TRUE Token ID for the true literal: 'true'
static int	UNKNOWN Token ID for an unknown token.

Constructor Summary

Token(SourceInfo info, int tokenID)

Method Summary

void	dump() Dumps the contents of this Token to System.out.
void	dump(PrintStream out) Dumps the contents of this Token.
static int	findReservedWordID(StringBuilder word) If the given StringBuilder starts with a valid token type, its ID is returned.
String	getCode() Token code is non-null, and is exactly the same as the name for its ID.
double	getDoubleValue() Only valid if token is a number.
float	getFloatValue() Only valid if token is a number.
int	getID() Returns the ID of this Token, which identifies what type of token it is.
String	getImage() Token image represents what a static token looks like in a source file.
int	getIntValue() Only valid if token is a number.
long	getLongValue() Only valid if token is a number.
static int	getMatchingBracket(int id) If the given id is a bracket: (,[,{,},],) then the matching bracket's id is returned.
int	getNumericType() Only valid if token is a number.
SourceInfo	getSourceInfo() Returns information regarding where in the source file this token came from.
String	getStringValue()
boolean	isBracket() Returns true if this Token is a bracket: (,[,{,},],)
static boolean	isBracket(int id) Returns true if id is a bracket
boolean	isCloseBracket() Returns true if this Token is a close bracket: ),],}
static boolean	isCloseBracket(int id) Returns true if id is a close bracket: ), ], }
boolean	isOpenBracket() Returns true if this Token is an open bracket: (,[,{
static boolean	isOpenBracket(int id) Returns true if id is an open bracket: (,[,{
boolean	isOperator() Returns true if this Token is an operator
static boolean	isOperator(int id) Returns true if id is an operator
boolean	isReservedWord() Returns true if this Token is a reserved word.
static boolean	isReservedWord(int id) Returns true if id is a reserved word
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

UNKNOWN

public static final int UNKNOWN

Token ID for an unknown token.

See Also:

Constant Field Values

EOF

public static final int EOF

Token ID for the end of file.

See Also:

Constant Field Values

COMMENT

public static final int COMMENT

Token ID for a single-line or multi-line comment.

See Also:

Constant Field Values

ENTER_CODE

public static final int ENTER_CODE

Token ID for the start of a code region.

See Also:

Constant Field Values

ENTER_TEXT

public static final int ENTER_TEXT

Token ID for the start of a text region.

See Also:

Constant Field Values

STRING

public static final int STRING

Token ID for a string literal.

See Also:

Constant Field Values

NUMBER

public static final int NUMBER

Token ID for a number literal.

See Also:

Constant Field Values

IDENT

public static final int IDENT

Token ID for an identifier.

See Also:

Constant Field Values

LPAREN

public static final int LPAREN

Token ID for the left parenthesis: '('

See Also:

Constant Field Values

RPAREN

public static final int RPAREN

Token ID for the right parenthesis: ')'

See Also:

Constant Field Values

LBRACE

public static final int LBRACE

Token ID for the left brace: '{'

See Also:

Constant Field Values

RBRACE

public static final int RBRACE

Token ID for the right brace: '}'

See Also:

Constant Field Values

LBRACK

public static final int LBRACK

Token ID for the left bracket: '['

See Also:

Constant Field Values

RBRACK

public static final int RBRACK

Token ID for the right bracket: ']'

See Also:

Constant Field Values

SEMI

public static final int SEMI

Token ID for the semi-colon: ';'

See Also:

Constant Field Values

COMMA

public static final int COMMA

Token ID for the comma: ','

See Also:

Constant Field Values

DOT

public static final int DOT

Token ID for the dot: '.'

See Also:

Constant Field Values

DOTDOT

public static final int DOTDOT

Token ID for the double dot: '..'

See Also:

Constant Field Values

ELLIPSIS

public static final int ELLIPSIS

Token ID for the ellipsis: '...'

See Also:

Constant Field Values

SLASH

public static final int SLASH

Token ID for the forward slash: '/'

See Also:

Constant Field Values

HASH

public static final int HASH

Token ID for the hash: '#'

See Also:

Constant Field Values

DOUBLE_HASH

public static final int DOUBLE_HASH

Token ID for the double hash: '##'

See Also:

Constant Field Values

LT

public static final int LT

Token ID for the less than operator: '<'

See Also:

Constant Field Values

LE

public static final int LE

Token ID for the less than or equal operator: '<='

See Also:

Constant Field Values

EQ

public static final int EQ

Token ID for the equal operator: '=='

See Also:

Constant Field Values

GE

public static final int GE

Token ID for the greater than or equal operator: '>='

See Also:

Constant Field Values

GT

public static final int GT

Token ID for the greater than operator: '>'

See Also:

Constant Field Values

NE

public static final int NE

Token ID for the not equal operator: '!='

See Also:

Constant Field Values

CONCAT

public static final int CONCAT

Token ID for the concatenation operator: '&'

See Also:

Constant Field Values

PLUS

public static final int PLUS

Token ID for the plus operator: '+'

See Also:

Constant Field Values

MINUS

public static final int MINUS

Token ID for the minus operator: '-'

See Also:

Constant Field Values

MULT

public static final int MULT

Token ID for the multiplication operator: '*'

See Also:

Constant Field Values

DIV

public static final int DIV

Token ID for the division operator: '/'

See Also:

Constant Field Values

MOD

public static final int MOD

Token ID for the modulus operator: '%'

See Also:

Constant Field Values

ASSIGN

public static final int ASSIGN

Token ID for the assignment operator: '='

See Also:

Constant Field Values

EQUAL_GREATER

public static final int EQUAL_GREATER

Token ID for equal greater: '=>'

See Also:

Constant Field Values

COLON

public static final int COLON

Token ID for colon: ':'

See Also:

Constant Field Values

QUESTION

public static final int QUESTION

Token ID for question mark: '?'

See Also:

Constant Field Values

LAMBDA

public static final int LAMBDA

Token ID for lambda operator: '->'

See Also:

Constant Field Values

SPACESHIP

public static final int SPACESHIP

Token ID for spaceship (comparator) operator: '<=>'

See Also:

Constant Field Values

SPREAD

public static final int SPREAD

Token ID for spread operator: *.

See Also:

Constant Field Values

NULL

public static final int NULL

Token ID for the null literal: 'null'

See Also:

Constant Field Values

TRUE

public static final int TRUE

Token ID for the true literal: 'true'

See Also:

Constant Field Values

FALSE

public static final int FALSE

Token ID for the false literal: 'false'

See Also:

Constant Field Values

NOT

public static final int NOT

Token ID for the not keyword: 'not'

See Also:

Constant Field Values

OR

public static final int OR

Token ID for the or keyword: 'or'

See Also:

Constant Field Values

AND

public static final int AND

Token ID for the and keyword: 'and'

See Also:

Constant Field Values

IF

public static final int IF

Token ID for the if keyword: 'if'

See Also:

Constant Field Values

ELSE

public static final int ELSE

Token ID for the else keyword: 'else'

See Also:

Constant Field Values

ISA

public static final int ISA

Token ID for the is-a keyword: 'isa'

See Also:

Constant Field Values

FOREACH

public static final int FOREACH

Token ID for the for-each keyword: 'foreach'

See Also:

Constant Field Values

IN

public static final int IN

Token ID for the in keyword: 'in'

See Also:

Constant Field Values

REVERSE

public static final int REVERSE

Token ID for the reverse keyword: 'reverse'

See Also:

Constant Field Values

TEMPLATE

public static final int TEMPLATE

Token ID for the template keyword: 'template'

See Also:

Constant Field Values

CALL

public static final int CALL

Token ID for the call keyword: 'call'

See Also:

Constant Field Values

BREAK

public static final int BREAK

Token ID for the break keyword: 'break'

See Also:

Constant Field Values

DEFINE

public static final int DEFINE

Token ID for the define keyword: 'define'

See Also:

Constant Field Values

AS

public static final int AS

Token ID for the as keyword: 'as'

See Also:

Constant Field Values

IMPORT

public static final int IMPORT

Token ID for the import keyword: 'import'

See Also:

Constant Field Values

CONTINUE

public static final int CONTINUE

Token ID for the continue keyword: 'continue'

See Also:

Constant Field Values

CLASS

public static final int CLASS

Token ID for the class keyword: 'class'

See Also:

Constant Field Values

Constructor Detail

Token

public Token(SourceInfo info,
             int tokenID)

Method Detail

isReservedWord

public static final boolean isReservedWord(int id)

Returns true if id is a reserved word

Parameters:

id - The Token id to test

isOperator

public static final boolean isOperator(int id)

Returns true if id is an operator

Parameters:

id - The Token id to test

isBracket

public static final boolean isBracket(int id)

Returns true if id is a bracket

Parameters:

id - The Token id to test

isOpenBracket

public static final boolean isOpenBracket(int id)

Returns true if id is an open bracket: (,[,{

Parameters:

id - The Token id to test

isCloseBracket

public static final boolean isCloseBracket(int id)

Returns true if id is a close bracket: ), ], }

Parameters:

id - The Token id to test

getMatchingBracket

public static final int getMatchingBracket(int id)

If the given id is a bracket: (,[,{,},],) then the matching bracket's id is returned. If id is not a bracket, then -1 is returned.

findReservedWordID

public static int findReservedWordID(StringBuilder word)

If the given StringBuilder starts with a valid token type, its ID is returned. Otherwise, the token ID UNKNOWN is returned.

dump

public final void dump()

Dumps the contents of this Token to System.out.

dump

public final void dump(PrintStream out)

Dumps the contents of this Token.

Parameters:

out - The PrintStream to write to.

getID

public final int getID()

Returns the ID of this Token, which identifies what type of token it is.

isReservedWord

public final boolean isReservedWord()

Returns true if this Token is a reserved word.

isBracket

public final boolean isBracket()

Returns true if this Token is a bracket: (,[,{,},],)

isOpenBracket

public final boolean isOpenBracket()

Returns true if this Token is an open bracket: (,[,{

isCloseBracket

public final boolean isCloseBracket()

Returns true if this Token is a close bracket: ),],}

isOperator

public final boolean isOperator()

Returns true if this Token is an operator

getImage

public String getImage()

Token image represents what a static token looks like in a source file. Token image is null if token is a string, number or identifier because these tokens don't have static images.

getCode

public String getCode()

Token code is non-null, and is exactly the same as the name for its ID.

getSourceInfo

public final SourceInfo getSourceInfo()

Returns information regarding where in the source file this token came from.

getStringValue

public String getStringValue()

getNumericType

public int getNumericType()

Only valid if token is a number. Returns 0 if token is not a number or is an invalid number. Returns 1 for int, 2 for long, 3 for float and 4 for double. The token ID for all numbers (even invalid ones) is NUMBER.

Returns:

0, 1, 2, 3 or 4.

getIntValue

public int getIntValue()

Only valid if token is a number.

getLongValue

public long getLongValue()

Only valid if token is a number.

getFloatValue

public float getFloatValue()

Only valid if token is a number.

getDoubleValue

public double getDoubleValue()

Only valid if token is a number.

toString

public String toString()

Overrides:

toString in class Object