parquet.hadoop

Class ParquetFileReader

java.lang.Object

parquet.hadoop.ParquetFileReader

All Implemented Interfaces:

Closeable, AutoCloseable

public class ParquetFileReader
extends Object
implements Closeable

Internal implementation of the Parquet file reader as a block container

Author:

Julien Le Dem

Constructor Summary

Constructors

Constructor and Description
ParquetFileReader(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path filePath, List<BlockMetaData> blocks, List<ColumnDescriptor> columns)

Method Summary

Methods

Modifier and Type	Method and Description
void	close()
static List<Footer>	readAllFootersInParallel(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.FileStatus fileStatus)
static List<Footer>	readAllFootersInParallel(org.apache.hadoop.conf.Configuration configuration, List<org.apache.hadoop.fs.FileStatus> partFiles)
static List<Footer>	readAllFootersInParallelUsingSummaryFiles(org.apache.hadoop.conf.Configuration configuration, List<org.apache.hadoop.fs.FileStatus> partFiles) for files provided, check if there's a summary file.
static ParquetMetadata	readFooter(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.FileStatus file) Reads the meta data block in the footer of the file
static ParquetMetadata	readFooter(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path file) Reads the meta data block in the footer of the file
static List<Footer>	readFooters(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.FileStatus pathStatus)
static List<Footer>	readFooters(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path file)
PageReadStore	readNextRowGroup() Reads all the columns requested from the row group at the current file position.
static List<Footer>	readSummaryFile(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.FileStatus summaryStatus)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ParquetFileReader

public ParquetFileReader(org.apache.hadoop.conf.Configuration configuration,
                 org.apache.hadoop.fs.Path filePath,
                 List<BlockMetaData> blocks,
                 List<ColumnDescriptor> columns)
                  throws IOException

Parameters:

f - the Parquet file (will be opened for read in this constructor)

blocks - the blocks to read

colums - the columns to read (their path)

codecClassName - the codec used to compress the blocks

Throws:

IOException - if the file can not be opened

Method Detail

readAllFootersInParallelUsingSummaryFiles

public static List<Footer> readAllFootersInParallelUsingSummaryFiles(org.apache.hadoop.conf.Configuration configuration,
                                                     List<org.apache.hadoop.fs.FileStatus> partFiles)
                                                              throws IOException

for files provided, check if there's a summary file. If a summary file is found it is used otherwise the file footer is used.

Parameters:

configuration - the hadoop conf to connect to the file system;

partFiles - the part files to read

Returns:

the footers for those files using the summary file if possible.

Throws:

IOException

readAllFootersInParallel

public static List<Footer> readAllFootersInParallel(org.apache.hadoop.conf.Configuration configuration,
                                    List<org.apache.hadoop.fs.FileStatus> partFiles)
                                             throws IOException

Throws:

IOException

readAllFootersInParallel

public static List<Footer> readAllFootersInParallel(org.apache.hadoop.conf.Configuration configuration,
                                    org.apache.hadoop.fs.FileStatus fileStatus)
                                             throws IOException

Throws:

IOException

readFooters

public static List<Footer> readFooters(org.apache.hadoop.conf.Configuration configuration,
                       org.apache.hadoop.fs.FileStatus pathStatus)
                                throws IOException

Throws:

IOException

readSummaryFile

public static List<Footer> readSummaryFile(org.apache.hadoop.conf.Configuration configuration,
                           org.apache.hadoop.fs.FileStatus summaryStatus)
                                    throws IOException

Throws:

IOException

readFooter

public static final ParquetMetadata readFooter(org.apache.hadoop.conf.Configuration configuration,
                         org.apache.hadoop.fs.Path file)
                                        throws IOException

Reads the meta data block in the footer of the file

Parameters:

configuration -

file - the parquet File

Returns:

the metadata blocks in the footer

Throws:

IOException - if an error occurs while reading the file

readFooters

public static final List<Footer> readFooters(org.apache.hadoop.conf.Configuration configuration,
                       org.apache.hadoop.fs.Path file)
                                      throws IOException

Throws:

IOException

readFooter

public static final ParquetMetadata readFooter(org.apache.hadoop.conf.Configuration configuration,
                         org.apache.hadoop.fs.FileStatus file)
                                        throws IOException

Reads the meta data block in the footer of the file

Parameters:

configuration -

file - the parquet File

Returns:

the metadata blocks in the footer

Throws:

IOException - if an error occurs while reading the file

readNextRowGroup

public PageReadStore readNextRowGroup()
                               throws IOException

Reads all the columns requested from the row group at the current file position.

Returns:

the PageReadStore which can provide PageReaders for each column.

Throws:

IOException - if an error occurs while reading

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException