com.github.jlangch.venice.impl.reader

Class Tokenizer

java.lang.Object

com.github.jlangch.venice.impl.reader.Tokenizer

public class Tokenizer
extends java.lang.Object

Method Summary

Modifier and Type	Method and Description
static java.util.List<Token>	tokenize(java.lang.String text, java.lang.String fileName)
static java.util.List<Token>	tokenize(java.lang.String text, java.lang.String fileName, boolean skipWhitespaces, boolean errorOnUnbalancedStringQuotes, boolean errorOnIncompleteEscapeChars)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

tokenize

public static java.util.List<Token> tokenize(java.lang.String text,
                                             java.lang.String fileName)

tokenize

public static java.util.List<Token> tokenize(java.lang.String text,
                                             java.lang.String fileName,
                                             boolean skipWhitespaces,
                                             boolean errorOnUnbalancedStringQuotes,
                                             boolean errorOnIncompleteEscapeChars)