water.parser.parquet

Class VecParquetReader

java.lang.Object

water.parser.parquet.VecParquetReader

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable

public class VecParquetReader
extends java.lang.Object
implements java.io.Closeable

Implementation of Parquet Reader working on H2O's Vecs. Note: This class was derived from Parquet's ParquetReader implementation. We cannot directly use the original implementation because it uses Hadoop FileSystem to access source data (and also Parquet summary files), it uses its own parallel implementation for reading metadata information which doesn't fit into H2O's architecture.

Constructor Summary

Constructors

Constructor and Description
VecParquetReader(water.fvec.Vec vec, org.apache.parquet.hadoop.metadata.ParquetMetadata metadata, water.parser.ParseWriter writer, byte[] chunkSchema)

Method Summary

Methods

Modifier and Type	Method and Description
void	close()
long	getInvalidRecordCount()
java.lang.Integer	read()
static org.apache.parquet.hadoop.metadata.ParquetMetadata	readFooter(byte[] metadataBytes)
static org.apache.parquet.hadoop.metadata.ParquetMetadata	readFooter(byte[] metadataBytes, org.apache.parquet.format.converter.ParquetMetadataConverter.MetadataFilter filter)
static byte[]	readFooterAsBytes(water.fvec.Vec vec)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

VecParquetReader

public VecParquetReader(water.fvec.Vec vec,
                org.apache.parquet.hadoop.metadata.ParquetMetadata metadata,
                water.parser.ParseWriter writer,
                byte[] chunkSchema)

Method Detail

read

public java.lang.Integer read()
                       throws java.io.IOException

Returns:

the index of added Chunk record or null if finished

Throws:

java.io.IOException

close

public void close()
           throws java.io.IOException

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface java.lang.AutoCloseable

Throws:

java.io.IOException

getInvalidRecordCount

public long getInvalidRecordCount()

readFooterAsBytes

public static byte[] readFooterAsBytes(water.fvec.Vec vec)

readFooter

public static org.apache.parquet.hadoop.metadata.ParquetMetadata readFooter(byte[] metadataBytes)

readFooter

public static org.apache.parquet.hadoop.metadata.ParquetMetadata readFooter(byte[] metadataBytes,
                                                            org.apache.parquet.format.converter.ParquetMetadataConverter.MetadataFilter filter)