com.github.jlangch.venice.impl.reader

Class Tokenizer

java.lang.Object

com.github.jlangch.venice.impl.reader.Tokenizer

public class Tokenizer
extends java.lang.Object

Method Summary

Modifier and Type	Method and Description
static java.util.List<Token>	tokenize(java.lang.String text, java.lang.String fileName)
static java.util.List<Token>	tokenize(java.lang.String text, java.lang.String fileName, boolean skipWhitespaces, boolean errorOnUnbalancedStringQuotes)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

tokenize

public static java.util.List<Token> tokenize(java.lang.String text,
                                             java.lang.String fileName)

tokenize

public static java.util.List<Token> tokenize(java.lang.String text,
                                             java.lang.String fileName,
                                             boolean skipWhitespaces,
                                             boolean errorOnUnbalancedStringQuotes)