org.apache.parquet.hadoop

Class ParquetFileReader

java.lang.Object

org.apache.parquet.hadoop.ParquetFileReader

All Implemented Interfaces:

Closeable, AutoCloseable

public class ParquetFileReader
extends Object
implements Closeable

Internal implementation of the Parquet file reader as a block container

Author:

Julien Le Dem

Field Summary

Fields

Modifier and Type	Field and Description
static String	PARQUET_READ_PARALLELISM

Constructor Summary

Constructors

Constructor and Description
ParquetFileReader(org.apache.hadoop.conf.Configuration configuration, FileMetaData fileMetaData, org.apache.hadoop.fs.Path filePath, List<BlockMetaData> blocks, List<ColumnDescriptor> columns) Deprecated.
ParquetFileReader(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path filePath, List<BlockMetaData> blocks, List<ColumnDescriptor> columns) Deprecated. use @link{ParquetFileReader(Configuration configuration, FileMetaData fileMetaData, Path filePath, List blocks, List columns)} instead
ParquetFileReader(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path file, ParquetMetadata footer)
ParquetFileReader(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path file, ParquetMetadataConverter.MetadataFilter filter)

Method Summary

Methods

Modifier and Type	Method and Description
void	appendTo(ParquetFileWriter writer)
void	close()
org.apache.parquet.hadoop.DictionaryPageReader	getDictionaryReader(BlockMetaData block)
FileMetaData	getFileMetaData()
ParquetMetadata	getFooter()
DictionaryPageReadStore	getNextDictionaryReader() Returns a DictionaryPageReadStore for the row group that would be returned by calling readNextRowGroup() or skipped by calling skipNextRowGroup().
org.apache.hadoop.fs.Path	getPath()
long	getRecordCount()
List<BlockMetaData>	getRowGroups()
static ParquetFileReader	open(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path file)
static ParquetFileReader	open(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path file, ParquetMetadata footer)
static ParquetFileReader	open(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path file, ParquetMetadataConverter.MetadataFilter filter)
static List<Footer>	readAllFootersInParallel(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.FileStatus fileStatus) Read the footers of all the files under that path (recursively) not using summary files.
static List<Footer>	readAllFootersInParallel(org.apache.hadoop.conf.Configuration configuration, List<org.apache.hadoop.fs.FileStatus> partFiles) Deprecated.
static List<Footer>	readAllFootersInParallel(org.apache.hadoop.conf.Configuration configuration, List<org.apache.hadoop.fs.FileStatus> partFiles, boolean skipRowGroups) read all the footers of the files provided (not using summary files)
static List<Footer>	readAllFootersInParallelUsingSummaryFiles(org.apache.hadoop.conf.Configuration configuration, Collection<org.apache.hadoop.fs.FileStatus> partFiles, boolean skipRowGroups) for files provided, check if there's a summary file.
static List<Footer>	readAllFootersInParallelUsingSummaryFiles(org.apache.hadoop.conf.Configuration configuration, List<org.apache.hadoop.fs.FileStatus> partFiles) Deprecated.
static ParquetMetadata	readFooter(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.FileStatus file) Deprecated. use ParquetFileReader#readFooter(Configuration, FileStatus, MetadataFilter)
static ParquetMetadata	readFooter(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.FileStatus file, ParquetMetadataConverter.MetadataFilter filter) Reads the meta data block in the footer of the file
static ParquetMetadata	readFooter(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path file) Deprecated.
static ParquetMetadata	readFooter(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path file, ParquetMetadataConverter.MetadataFilter filter) Reads the meta data in the footer of the file.
static ParquetMetadata	readFooter(InputFile file, ParquetMetadataConverter.MetadataFilter filter) Reads the meta data block in the footer of the file using provided input stream
static List<Footer>	readFooters(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.FileStatus pathStatus) Deprecated.
static List<Footer>	readFooters(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.FileStatus pathStatus, boolean skipRowGroups) Read the footers of all the files under that path (recursively) using summary files if possible
static List<Footer>	readFooters(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path path) Deprecated.
PageReadStore	readNextRowGroup() Reads all the columns requested from the row group at the current file position.
static List<Footer>	readSummaryFile(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.FileStatus summaryStatus) Specifically reads a given summary file
void	setRequestedSchema(MessageType projection)
boolean	skipNextRowGroup()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PARQUET_READ_PARALLELISM

public static String PARQUET_READ_PARALLELISM

Constructor Detail

ParquetFileReader

public ParquetFileReader(org.apache.hadoop.conf.Configuration configuration,
                 org.apache.hadoop.fs.Path filePath,
                 List<BlockMetaData> blocks,
                 List<ColumnDescriptor> columns)
                  throws IOException

Deprecated. use @link{ParquetFileReader(Configuration configuration, FileMetaData fileMetaData, Path filePath, List blocks, List columns)} instead

Throws:

IOException

ParquetFileReader

@Deprecated
public ParquetFileReader(org.apache.hadoop.conf.Configuration configuration,
                            FileMetaData fileMetaData,
                            org.apache.hadoop.fs.Path filePath,
                            List<BlockMetaData> blocks,
                            List<ColumnDescriptor> columns)
                  throws IOException

Deprecated.

Parameters:

configuration - the Hadoop conf

fileMetaData - fileMetaData for parquet file

blocks - the blocks to read

columns - the columns to read (their path)

Throws:

IOException - if the file can not be opened

ParquetFileReader

public ParquetFileReader(org.apache.hadoop.conf.Configuration conf,
                 org.apache.hadoop.fs.Path file,
                 ParquetMetadataConverter.MetadataFilter filter)
                  throws IOException

Parameters:

conf - the Hadoop Configuration

file - Path to a parquet file

filter - a ParquetMetadataConverter.MetadataFilter for selecting row groups

Throws:

IOException - if the file can not be opened

ParquetFileReader

public ParquetFileReader(org.apache.hadoop.conf.Configuration conf,
                 org.apache.hadoop.fs.Path file,
                 ParquetMetadata footer)
                  throws IOException

Parameters:

conf - the Hadoop Configuration

file - Path to a parquet file

footer - a ParquetMetadata footer already read from the file

Throws:

IOException - if the file can not be opened

Method Detail

readAllFootersInParallelUsingSummaryFiles

@Deprecated
public static List<Footer> readAllFootersInParallelUsingSummaryFiles(org.apache.hadoop.conf.Configuration configuration,
                                                                List<org.apache.hadoop.fs.FileStatus> partFiles)
                                                              throws IOException

Deprecated.

for files provided, check if there's a summary file. If a summary file is found it is used otherwise the file footer is used.

Parameters:

configuration - the hadoop conf to connect to the file system;

partFiles - the part files to read

Returns:

the footers for those files using the summary file if possible.

Throws:

IOException

readAllFootersInParallelUsingSummaryFiles

public static List<Footer> readAllFootersInParallelUsingSummaryFiles(org.apache.hadoop.conf.Configuration configuration,
                                                     Collection<org.apache.hadoop.fs.FileStatus> partFiles,
                                                     boolean skipRowGroups)
                                                              throws IOException

for files provided, check if there's a summary file. If a summary file is found it is used otherwise the file footer is used.

Parameters:

configuration - the hadoop conf to connect to the file system;

partFiles - the part files to read

skipRowGroups - to skipRowGroups in the footers

Returns:

the footers for those files using the summary file if possible.

Throws:

IOException

readAllFootersInParallel

@Deprecated
public static List<Footer> readAllFootersInParallel(org.apache.hadoop.conf.Configuration configuration,
                                               List<org.apache.hadoop.fs.FileStatus> partFiles)
                                             throws IOException

Deprecated.

Throws:

IOException

readAllFootersInParallel

public static List<Footer> readAllFootersInParallel(org.apache.hadoop.conf.Configuration configuration,
                                    List<org.apache.hadoop.fs.FileStatus> partFiles,
                                    boolean skipRowGroups)
                                             throws IOException

read all the footers of the files provided (not using summary files)

Parameters:

configuration - the conf to access the File System

partFiles - the files to read

skipRowGroups - to skip the rowGroup info

Returns:

the footers

Throws:

IOException

readAllFootersInParallel

public static List<Footer> readAllFootersInParallel(org.apache.hadoop.conf.Configuration configuration,
                                    org.apache.hadoop.fs.FileStatus fileStatus)
                                             throws IOException

Read the footers of all the files under that path (recursively) not using summary files. rowGroups are not skipped

Parameters:

configuration - the configuration to access the FS

fileStatus - the root dir

Returns:

all the footers

Throws:

IOException

readFooters

@Deprecated
public static List<Footer> readFooters(org.apache.hadoop.conf.Configuration configuration,
                                  org.apache.hadoop.fs.Path path)
                                throws IOException

Deprecated.

Throws:

IOException

readFooters

@Deprecated
public static List<Footer> readFooters(org.apache.hadoop.conf.Configuration configuration,
                                  org.apache.hadoop.fs.FileStatus pathStatus)
                                throws IOException

Deprecated.

this always returns the row groups

Parameters:

configuration -

pathStatus -

Returns:

Throws:

IOException

readFooters

public static List<Footer> readFooters(org.apache.hadoop.conf.Configuration configuration,
                       org.apache.hadoop.fs.FileStatus pathStatus,
                       boolean skipRowGroups)
                                throws IOException

Read the footers of all the files under that path (recursively) using summary files if possible

Parameters:

configuration - the configuration to access the FS

fileStatus - the root dir

Returns:

all the footers

Throws:

IOException

readSummaryFile

public static List<Footer> readSummaryFile(org.apache.hadoop.conf.Configuration configuration,
                           org.apache.hadoop.fs.FileStatus summaryStatus)
                                    throws IOException

Specifically reads a given summary file

Parameters:

configuration -

summaryStatus -

Returns:

the metadata translated for each file

Throws:

IOException

readFooter

@Deprecated
public static final ParquetMetadata readFooter(org.apache.hadoop.conf.Configuration configuration,
                                    org.apache.hadoop.fs.Path file)
                                        throws IOException

Deprecated.

Reads the meta data block in the footer of the file

Parameters:

configuration -

file - the parquet File

Returns:

the metadata blocks in the footer

Throws:

IOException - if an error occurs while reading the file

readFooter

public static ParquetMetadata readFooter(org.apache.hadoop.conf.Configuration configuration,
                         org.apache.hadoop.fs.Path file,
                         ParquetMetadataConverter.MetadataFilter filter)
                                  throws IOException

Reads the meta data in the footer of the file. Skipping row groups (or not) based on the provided filter

Parameters:

configuration -

file - the Parquet File

filter - the filter to apply to row groups

Returns:

the metadata with row groups filtered.

Throws:

IOException - if an error occurs while reading the file

readFooter

@Deprecated
public static final ParquetMetadata readFooter(org.apache.hadoop.conf.Configuration configuration,
                                    org.apache.hadoop.fs.FileStatus file)
                                        throws IOException

Deprecated. use ParquetFileReader#readFooter(Configuration, FileStatus, MetadataFilter)

Throws:

IOException

readFooter

public static final ParquetMetadata readFooter(org.apache.hadoop.conf.Configuration configuration,
                         org.apache.hadoop.fs.FileStatus file,
                         ParquetMetadataConverter.MetadataFilter filter)
                                        throws IOException

Reads the meta data block in the footer of the file

Parameters:

configuration -

file - the parquet File

filter - the filter to apply to row groups

Returns:

the metadata blocks in the footer

Throws:

IOException - if an error occurs while reading the file

readFooter

public static final ParquetMetadata readFooter(InputFile file,
                         ParquetMetadataConverter.MetadataFilter filter)
                                        throws IOException

Reads the meta data block in the footer of the file using provided input stream

Parameters:

file - a InputFile to read

filter - the filter to apply to row groups

Returns:

the metadata blocks in the footer

Throws:

IOException - if an error occurs while reading the file

open

public static ParquetFileReader open(org.apache.hadoop.conf.Configuration conf,
                     org.apache.hadoop.fs.Path file)
                              throws IOException

Throws:

IOException

open

public static ParquetFileReader open(org.apache.hadoop.conf.Configuration conf,
                     org.apache.hadoop.fs.Path file,
                     ParquetMetadataConverter.MetadataFilter filter)
                              throws IOException

Throws:

IOException

open

public static ParquetFileReader open(org.apache.hadoop.conf.Configuration conf,
                     org.apache.hadoop.fs.Path file,
                     ParquetMetadata footer)
                              throws IOException

Throws:

IOException

getFooter

public ParquetMetadata getFooter()

getFileMetaData

public FileMetaData getFileMetaData()

getRecordCount

public long getRecordCount()

getPath

public org.apache.hadoop.fs.Path getPath()

getRowGroups

public List<BlockMetaData> getRowGroups()

setRequestedSchema

public void setRequestedSchema(MessageType projection)

appendTo

public void appendTo(ParquetFileWriter writer)
              throws IOException

Throws:

IOException

readNextRowGroup

public PageReadStore readNextRowGroup()
                               throws IOException

Reads all the columns requested from the row group at the current file position.

Returns:

the PageReadStore which can provide PageReaders for each column.

Throws:

IOException - if an error occurs while reading

skipNextRowGroup

public boolean skipNextRowGroup()

getNextDictionaryReader

public DictionaryPageReadStore getNextDictionaryReader()

Returns a DictionaryPageReadStore for the row group that would be returned by calling readNextRowGroup() or skipped by calling skipNextRowGroup().

Returns:

a DictionaryPageReadStore for the next row group

getDictionaryReader

public org.apache.parquet.hadoop.DictionaryPageReader getDictionaryReader(BlockMetaData block)

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException