org.dmd.util.parsing

Class Token

java.lang.Object

org.dmd.util.parsing.Token

public class Token
extends Object

The Token class is used to represent a string fragment and a type designation. The type designation is usually assigned by a parsing algorithm, such as that used in the Classifier class; see the Classifier description for more details.

Token defines a set of constant properties that indicate basic token types. If you wish to define your own identifiers for tokens, you should start at Token.CUSTOM + 1.

See Also:

Classifier

Field Summary

Fields

Modifier and Type	Field and Description
static int	CUSTOM CUSTOM is simply an indicator of where you should start assigning identifiers to your own custom tokens.
static int	DQUOTE DQUOTE indicates that the token is a double quote character.
static int	EMPTY EMPTY indicates an empty or unassigned token.
static int	FLOAT FLOAT indicates that the token is an floating point number i.e.
static int	IGNORED IGNORED indicates that the token should be ignored during parsing.
static int	INTEGER INTEGER indicates that the token is an integer.
static int	QUOTEDTEXT QUOTEDTEXT indicates that the token represents a chunk of text that was enclosed in double quotes.
static int	TEXT TEXT indicates that the token is just a piece of text as opposed to a number.

Constructor Summary

Constructors

Constructor and Description
Token(String v, int t) Creates a new token of the specified type.
Token(String v, int t, Object u) Creates a new token of the specified type.
Token(Token t) Creates a new token that's a copy of the supplied token.

Method Summary

Modifier and Type	Method and Description
int	getPos() Sets the position.
int	getType() Returns the classified type of the token.
Object	getUserData() Returns the userdata associated with the token.
String	getValue() Returns the string value of the token.
boolean	isKeyWord()
void	setPos(int p) Sets the position.
void	setUserData(Object o) Sets the userdata.
String	toString() Formats the token as a String.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

EMPTY

public static final int EMPTY

EMPTY indicates an empty or unassigned token.

See Also:

Constant Field Values

IGNORED

public static final int IGNORED

IGNORED indicates that the token should be ignored during parsing.

See Also:

Constant Field Values

TEXT

public static final int TEXT

TEXT indicates that the token is just a piece of text as opposed to a number.

See Also:

Constant Field Values

QUOTEDTEXT

public static final int QUOTEDTEXT

QUOTEDTEXT indicates that the token represents a chunk of text that was enclosed in double quotes.

See Also:

Constant Field Values

INTEGER

public static final int INTEGER

INTEGER indicates that the token is an integer.

See Also:

Constant Field Values

FLOAT

public static final int FLOAT

FLOAT indicates that the token is an floating point number i.e. with embedded decimal point.

See Also:

Constant Field Values

DQUOTE

public static final int DQUOTE

DQUOTE indicates that the token is a double quote character. This is included here to facilitate the representation of QUOTEDTEXT.

See Also:

Constant Field Values

CUSTOM

public static final int CUSTOM

CUSTOM is simply an indicator of where you should start assigning identifiers to your own custom tokens. This prevents clashing of the identifier space.

See Also:

Constant Field Values

Constructor Detail

Token

public Token(String v,
             int t)

Creates a new token of the specified type.

Parameters:

v - The string value of the token.

t - The type indicator.

Token

public Token(String v,
             int t,
             Object u)

Creates a new token of the specified type.

Parameters:

v - The string value of the token.

t - The type indicator.

u - The user data.

Token

public Token(Token t)

Creates a new token that's a copy of the supplied token.

Parameters:

t - The existing token.

Method Detail

getValue

public String getValue()

Returns the string value of the token.

getType

public int getType()

Returns the classified type of the token.

getUserData

public Object getUserData()

Returns the userdata associated with the token.

setPos

public void setPos(int p)

Sets the position.

getPos

public int getPos()

Sets the position.

setUserData

public void setUserData(Object o)

Sets the userdata.

toString

public String toString()

Formats the token as a String. The format is \n

Overrides:

toString in class Object

isKeyWord

public boolean isKeyWord()