water.parser.orc

Class OrcParser

java.lang.Object

water.Iced

water.parser.Parser

water.parser.orc.OrcParser

All Implemented Interfaces:

java.io.Externalizable, java.io.Serializable, java.lang.Cloneable, water.Freezable

public class OrcParser
extends water.parser.Parser

ORC parser for H2O distributed parsing subsystem. Basically, here is the plan: To parse an Orc file, we need to do the following in order to get the following useful information: 1. Get a Reader rdr. 2. From the reader rdr, we can get the following pieces of information: a. number of columns, column types and column names. We only support parsing of primitive types; b. Lists of StripeInformation that describes how many stripes of data that we will need to read; c. For each stripe, get information like rows per stripe, data size in bytes 3. The plan is to read the file in parallel in whole numbers of stripes. 4. Inside each stripe, we will read data out in batches of VectorizedRowBatch (1024 rows or less).

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	OrcParser.OrcParseSetup

Field Summary

Fields

Modifier and Type	Field and Description
static int	ADD_OFFSET
static int	DAY_TO_MS
static int	HOUR_OFFSET

Fields inherited from class water.parser.Parser

_jobKey, _keepColumns, _setup, CHAR_DECIMAL_SEP, CHAR_SEPARATOR, COND_QUOTE, COND_QUOTED_NUMBER_END, COND_QUOTED_TOKEN, EOL, EXPECT_COND_LF, HASHTAG, LARGEST_DIGIT_NUMBER, NUMBER, NUMBER_END, NUMBER_EXP, NUMBER_EXP_START, NUMBER_FRACTION, NUMBER_SKIP, NUMBER_SKIP_NO_DOT, POSSIBLE_CURRENCY, POSSIBLE_EMPTY_LINE, POSSIBLE_ESCAPED_QUOTE, SEPARATOR_OR_EOL, SKIP_LINE, STRING, STRING_END, TOKEN, WHITESPACE_BEFORE_TOKEN

Method Summary

Modifier and Type	Method and Description
protected water.parser.ParseWriter	parseChunk(int chunkId, water.parser.ParseReader din, water.parser.ParseWriter dout) This method calculates the number of stripes that will be read for each chunk.
protected water.parser.ParseWriter	streamParse(java.io.InputStream is, water.parser.StreamParseWriter dout)
protected water.parser.ParseWriter	streamParseZip(java.io.InputStream is, water.parser.StreamParseWriter dout, java.io.InputStream bvs)

Methods inherited from class water.parser.Parser

fileHasHeader, isEOL, sequentialParse

Methods inherited from class water.Iced

asBytes, clone, copyOver, frozenType, read, readExternal, readJSON, reloadFromBytes, toJsonString, write, writeExternal, writeJSON

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DAY_TO_MS

public static final int DAY_TO_MS

See Also:

Constant Field Values

ADD_OFFSET

public static final int ADD_OFFSET

See Also:

Constant Field Values

HOUR_OFFSET

public static final int HOUR_OFFSET

See Also:

Constant Field Values

Method Detail

streamParse

protected water.parser.ParseWriter streamParse(java.io.InputStream is,
                                               water.parser.StreamParseWriter dout)
                                        throws java.io.IOException

Overrides:

streamParse in class water.parser.Parser

Throws:

java.io.IOException

streamParseZip

protected water.parser.ParseWriter streamParseZip(java.io.InputStream is,
                                                  water.parser.StreamParseWriter dout,
                                                  java.io.InputStream bvs)
                                           throws java.io.IOException

Overrides:

streamParseZip in class water.parser.Parser

Throws:

java.io.IOException

parseChunk

protected final water.parser.ParseWriter parseChunk(int chunkId,
                                                    water.parser.ParseReader din,
                                                    water.parser.ParseWriter dout)

This method calculates the number of stripes that will be read for each chunk. Since only single threading is supported in reading each stripe, we will never split one stripe over different chunks.

Specified by:

parseChunk in class water.parser.Parser

Parameters:

chunkId: - chunk index, calculated as file size/chunk size. The file size is calculated with data plus overhead in terms of headers and other info, number of chunks calculated will be higher than the actual chunks needed. If the chunk number is too high, the method will return without writing to dout.

din: - ParseReader, not used for parsing orc files

dout: - ParseWriter, used to add data to H2O frame.