org.apache.parquet.hadoop

Class ParquetFileReader

java.lang.Object

org.apache.parquet.hadoop.ParquetFileReader

All Implemented Interfaces:

Closeable, AutoCloseable

public class ParquetFileReader
extends Object
implements Closeable

Internal implementation of the Parquet file reader as a block container

Field Summary

Fields

Modifier and Type	Field and Description
static String	PARQUET_READ_PARALLELISM

Constructor Summary

Constructors

Constructor and Description
ParquetFileReader(org.apache.hadoop.conf.Configuration configuration, FileMetaData fileMetaData, org.apache.hadoop.fs.Path filePath, List<BlockMetaData> blocks, List<ColumnDescriptor> columns) Deprecated. will be removed in 2.0.0.
ParquetFileReader(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path filePath, List<BlockMetaData> blocks, List<ColumnDescriptor> columns) Deprecated. will be removed in 2.0.0.
ParquetFileReader(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path file, ParquetMetadata footer) Deprecated. will be removed in 2.0.0.
ParquetFileReader(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path file, ParquetMetadataConverter.MetadataFilter filter) Deprecated. will be removed in 2.0.0.
ParquetFileReader(InputFile file, ParquetReadOptions options)

Method Summary

Modifier and Type	Method and Description
void	appendTo(ParquetFileWriter writer)
void	close()
org.apache.parquet.hadoop.DictionaryPageReader	getDictionaryReader(BlockMetaData block)
String	getFile()
FileMetaData	getFileMetaData()
long	getFilteredRecordCount()
ParquetMetadata	getFooter()
DictionaryPageReadStore	getNextDictionaryReader() Returns a DictionaryPageReadStore for the row group that would be returned by calling readNextRowGroup() or skipped by calling skipNextRowGroup().
org.apache.hadoop.fs.Path	getPath() Deprecated. will be removed in 2.0.0; use getFile() instead
long	getRecordCount()
List<BlockMetaData>	getRowGroups()
static ParquetFileReader	open(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path file) Deprecated. will be removed in 2.0.0; use open(InputFile)
static ParquetFileReader	open(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path file, ParquetMetadata footer) Deprecated. will be removed in 2.0.0
static ParquetFileReader	open(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path file, ParquetMetadataConverter.MetadataFilter filter) Deprecated. will be removed in 2.0.0; use open(InputFile,ParquetReadOptions)
static ParquetFileReader	open(InputFile file) Open a file.
static ParquetFileReader	open(InputFile file, ParquetReadOptions options) Open a file with options.
static List<Footer>	readAllFootersInParallel(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.FileStatus fileStatus) Deprecated. will be removed in 2.0.0; use open(InputFile, ParquetReadOptions)
static List<Footer>	readAllFootersInParallel(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.FileStatus fileStatus, boolean skipRowGroups) Deprecated. will be removed in 2.0.0; use open(InputFile, ParquetReadOptions)
static List<Footer>	readAllFootersInParallel(org.apache.hadoop.conf.Configuration configuration, List<org.apache.hadoop.fs.FileStatus> partFiles) Deprecated. metadata files are not recommended and will be removed in 2.0.0
static List<Footer>	readAllFootersInParallel(org.apache.hadoop.conf.Configuration configuration, List<org.apache.hadoop.fs.FileStatus> partFiles, boolean skipRowGroups) Deprecated. will be removed in 2.0.0; use open(InputFile, ParquetReadOptions)
static List<Footer>	readAllFootersInParallelUsingSummaryFiles(org.apache.hadoop.conf.Configuration configuration, Collection<org.apache.hadoop.fs.FileStatus> partFiles, boolean skipRowGroups) Deprecated. metadata files are not recommended and will be removed in 2.0.0
static List<Footer>	readAllFootersInParallelUsingSummaryFiles(org.apache.hadoop.conf.Configuration configuration, List<org.apache.hadoop.fs.FileStatus> partFiles) Deprecated. metadata files are not recommended and will be removed in 2.0.0
ColumnIndex	readColumnIndex(ColumnChunkMetaData column)
static ParquetMetadata	readFooter(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.FileStatus file) Deprecated. will be removed in 2.0.0; use open(InputFile, ParquetReadOptions)
static ParquetMetadata	readFooter(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.FileStatus file, ParquetMetadataConverter.MetadataFilter filter) Deprecated. will be removed in 2.0.0; use open(InputFile, ParquetReadOptions)
static ParquetMetadata	readFooter(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path file) Deprecated. will be removed in 2.0.0; use open(InputFile, ParquetReadOptions)
static ParquetMetadata	readFooter(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path file, ParquetMetadataConverter.MetadataFilter filter) Deprecated. will be removed in 2.0.0; use open(InputFile, ParquetReadOptions)
static ParquetMetadata	readFooter(InputFile file, ParquetMetadataConverter.MetadataFilter filter) Deprecated. will be removed in 2.0.0; use open(InputFile, ParquetReadOptions)
static List<Footer>	readFooters(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.FileStatus pathStatus) Deprecated. will be removed in 2.0.0; use open(InputFile, ParquetReadOptions)
static List<Footer>	readFooters(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.FileStatus pathStatus, boolean skipRowGroups) Deprecated. will be removed in 2.0.0; use open(InputFile, ParquetReadOptions)
static List<Footer>	readFooters(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path path) Deprecated. will be removed in 2.0.0; use open(InputFile, ParquetReadOptions)
PageReadStore	readNextFilteredRowGroup() Reads all the columns requested from the row group at the current file position.
PageReadStore	readNextRowGroup() Reads all the columns requested from the row group at the current file position.
OffsetIndex	readOffsetIndex(ColumnChunkMetaData column)
static List<Footer>	readSummaryFile(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.FileStatus summaryStatus) Deprecated. metadata files are not recommended and will be removed in 2.0.0
void	setRequestedSchema(MessageType projection)
boolean	skipNextRowGroup()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PARQUET_READ_PARALLELISM

public static String PARQUET_READ_PARALLELISM

Constructor Detail

ParquetFileReader

@Deprecated
public ParquetFileReader(org.apache.hadoop.conf.Configuration configuration,
                                     org.apache.hadoop.fs.Path filePath,
                                     List<BlockMetaData> blocks,
                                     List<ColumnDescriptor> columns)
                              throws IOException

Deprecated. will be removed in 2.0.0.

Parameters:

configuration - the Hadoop conf

filePath - Path for the parquet file

blocks - the blocks to read

columns - the columns to read (their path)

Throws:

IOException - if the file can not be opened

ParquetFileReader

@Deprecated
public ParquetFileReader(org.apache.hadoop.conf.Configuration configuration,
                                     FileMetaData fileMetaData,
                                     org.apache.hadoop.fs.Path filePath,
                                     List<BlockMetaData> blocks,
                                     List<ColumnDescriptor> columns)
                              throws IOException

Deprecated. will be removed in 2.0.0.

Parameters:

configuration - the Hadoop conf

fileMetaData - fileMetaData for parquet file

filePath - Path for the parquet file

blocks - the blocks to read

columns - the columns to read (their path)

Throws:

IOException - if the file can not be opened

ParquetFileReader

@Deprecated
public ParquetFileReader(org.apache.hadoop.conf.Configuration conf,
                                     org.apache.hadoop.fs.Path file,
                                     ParquetMetadataConverter.MetadataFilter filter)
                              throws IOException

Deprecated. will be removed in 2.0.0.

Parameters:

conf - the Hadoop Configuration

file - Path to a parquet file

filter - a ParquetMetadataConverter.MetadataFilter for selecting row groups

Throws:

IOException - if the file can not be opened

ParquetFileReader

@Deprecated
public ParquetFileReader(org.apache.hadoop.conf.Configuration conf,
                                     org.apache.hadoop.fs.Path file,
                                     ParquetMetadata footer)
                              throws IOException

Deprecated. will be removed in 2.0.0.

Parameters:

conf - the Hadoop Configuration

file - Path to a parquet file

footer - a ParquetMetadata footer already read from the file

Throws:

IOException - if the file can not be opened

ParquetFileReader

public ParquetFileReader(InputFile file,
                         ParquetReadOptions options)
                  throws IOException

Throws:

IOException

Method Detail

readAllFootersInParallelUsingSummaryFiles

@Deprecated
public static List<Footer> readAllFootersInParallelUsingSummaryFiles(org.apache.hadoop.conf.Configuration configuration,
                                                                                 List<org.apache.hadoop.fs.FileStatus> partFiles)
                                                                          throws IOException

Deprecated. metadata files are not recommended and will be removed in 2.0.0

for files provided, check if there's a summary file. If a summary file is found it is used otherwise the file footer is used.

Parameters:

configuration - the hadoop conf to connect to the file system;

partFiles - the part files to read

Returns:

the footers for those files using the summary file if possible.

Throws:

IOException - if there is an exception while reading footers

readAllFootersInParallelUsingSummaryFiles

@Deprecated
public static List<Footer> readAllFootersInParallelUsingSummaryFiles(org.apache.hadoop.conf.Configuration configuration,
                                                                                 Collection<org.apache.hadoop.fs.FileStatus> partFiles,
                                                                                 boolean skipRowGroups)
                                                                          throws IOException

Deprecated. metadata files are not recommended and will be removed in 2.0.0

for files provided, check if there's a summary file. If a summary file is found it is used otherwise the file footer is used.

Parameters:

configuration - the hadoop conf to connect to the file system;

partFiles - the part files to read

skipRowGroups - to skipRowGroups in the footers

Returns:

the footers for those files using the summary file if possible.

Throws:

IOException - if there is an exception while reading footers

readAllFootersInParallel

@Deprecated
public static List<Footer> readAllFootersInParallel(org.apache.hadoop.conf.Configuration configuration,
                                                                List<org.apache.hadoop.fs.FileStatus> partFiles)
                                                         throws IOException

Deprecated. metadata files are not recommended and will be removed in 2.0.0

Parameters:

configuration - the conf to access the File System

partFiles - the files to read

Returns:

the footers

Throws:

IOException - if an exception was raised while reading footers

readAllFootersInParallel

@Deprecated
public static List<Footer> readAllFootersInParallel(org.apache.hadoop.conf.Configuration configuration,
                                                                List<org.apache.hadoop.fs.FileStatus> partFiles,
                                                                boolean skipRowGroups)
                                                         throws IOException

Deprecated. will be removed in 2.0.0; use open(InputFile, ParquetReadOptions)

read all the footers of the files provided (not using summary files)

Parameters:

configuration - the conf to access the File System

partFiles - the files to read

skipRowGroups - to skip the rowGroup info

Returns:

the footers

Throws:

IOException - if there is an exception while reading footers

readAllFootersInParallel

@Deprecated
public static List<Footer> readAllFootersInParallel(org.apache.hadoop.conf.Configuration configuration,
                                                                org.apache.hadoop.fs.FileStatus fileStatus,
                                                                boolean skipRowGroups)
                                                         throws IOException

Deprecated. will be removed in 2.0.0; use open(InputFile, ParquetReadOptions)

Read the footers of all the files under that path (recursively) not using summary files.

Parameters:

configuration - a configuration

fileStatus - a file status to recursively list

skipRowGroups - whether to skip reading row group metadata

Returns:

a list of footers

Throws:

IOException - if an exception is thrown while reading the footers

readAllFootersInParallel

@Deprecated
public static List<Footer> readAllFootersInParallel(org.apache.hadoop.conf.Configuration configuration,
                                                                org.apache.hadoop.fs.FileStatus fileStatus)
                                                         throws IOException

Deprecated. will be removed in 2.0.0; use open(InputFile, ParquetReadOptions)

Read the footers of all the files under that path (recursively) not using summary files. rowGroups are not skipped

Parameters:

configuration - the configuration to access the FS

fileStatus - the root dir

Returns:

all the footers

Throws:

IOException - if an exception is thrown while reading the footers

readFooters

@Deprecated
public static List<Footer> readFooters(org.apache.hadoop.conf.Configuration configuration,
                                                   org.apache.hadoop.fs.Path path)
                                            throws IOException

Deprecated. will be removed in 2.0.0; use open(InputFile, ParquetReadOptions)

Parameters:

configuration - a configuration

path - a file path

Returns:

a list of footers

Throws:

IOException - if an exception is thrown while reading the footers

readFooters

@Deprecated
public static List<Footer> readFooters(org.apache.hadoop.conf.Configuration configuration,
                                                   org.apache.hadoop.fs.FileStatus pathStatus)
                                            throws IOException

Deprecated. will be removed in 2.0.0; use open(InputFile, ParquetReadOptions)

this always returns the row groups

Parameters:

configuration - a configuration

pathStatus - a file status to read footers from

Returns:

a list of footers

Throws:

IOException - if an exception is thrown while reading the footers

readFooters

@Deprecated
public static List<Footer> readFooters(org.apache.hadoop.conf.Configuration configuration,
                                                   org.apache.hadoop.fs.FileStatus pathStatus,
                                                   boolean skipRowGroups)
                                            throws IOException

Deprecated. will be removed in 2.0.0; use open(InputFile, ParquetReadOptions)

Read the footers of all the files under that path (recursively) using summary files if possible

Parameters:

configuration - the configuration to access the FS

pathStatus - the root dir

skipRowGroups - whether to skip reading row group metadata

Returns:

all the footers

Throws:

IOException - if an exception is thrown while reading the footers

readSummaryFile

@Deprecated
public static List<Footer> readSummaryFile(org.apache.hadoop.conf.Configuration configuration,
                                                       org.apache.hadoop.fs.FileStatus summaryStatus)
                                                throws IOException

Deprecated. metadata files are not recommended and will be removed in 2.0.0

Specifically reads a given summary file

Parameters:

configuration - a configuration

summaryStatus - file status for a summary file

Returns:

the metadata translated for each file

Throws:

IOException - if an exception is thrown while reading the summary file

readFooter

@Deprecated
public static final ParquetMetadata readFooter(org.apache.hadoop.conf.Configuration configuration,
                                                           org.apache.hadoop.fs.Path file)
                                                    throws IOException

Deprecated. will be removed in 2.0.0; use open(InputFile, ParquetReadOptions)

Reads the meta data block in the footer of the file

Parameters:

configuration - a configuration

file - the parquet File

Returns:

the metadata blocks in the footer

Throws:

IOException - if an error occurs while reading the file

readFooter

public static ParquetMetadata readFooter(org.apache.hadoop.conf.Configuration configuration,
                                         org.apache.hadoop.fs.Path file,
                                         ParquetMetadataConverter.MetadataFilter filter)
                                  throws IOException

Deprecated. will be removed in 2.0.0; use open(InputFile, ParquetReadOptions)

Reads the meta data in the footer of the file. Skipping row groups (or not) based on the provided filter

Parameters:

configuration - a configuration

file - the Parquet File

filter - the filter to apply to row groups

Returns:

the metadata with row groups filtered.

Throws:

IOException - if an error occurs while reading the file

readFooter

@Deprecated
public static final ParquetMetadata readFooter(org.apache.hadoop.conf.Configuration configuration,
                                                           org.apache.hadoop.fs.FileStatus file)
                                                    throws IOException

Deprecated. will be removed in 2.0.0; use open(InputFile, ParquetReadOptions)

Parameters:

configuration - a configuration

file - the Parquet File

Returns:

the metadata with row groups.

Throws:

IOException - if an error occurs while reading the file

readFooter

@Deprecated
public static final ParquetMetadata readFooter(org.apache.hadoop.conf.Configuration configuration,
                                                           org.apache.hadoop.fs.FileStatus file,
                                                           ParquetMetadataConverter.MetadataFilter filter)
                                                    throws IOException

Deprecated. will be removed in 2.0.0; use open(InputFile, ParquetReadOptions)

Reads the meta data block in the footer of the file

Parameters:

configuration - a configuration

file - the parquet File

filter - the filter to apply to row groups

Returns:

the metadata blocks in the footer

Throws:

IOException - if an error occurs while reading the file

readFooter

@Deprecated
public static final ParquetMetadata readFooter(InputFile file,
                                                           ParquetMetadataConverter.MetadataFilter filter)
                                                    throws IOException

Deprecated. will be removed in 2.0.0; use open(InputFile, ParquetReadOptions)

Reads the meta data block in the footer of the file using provided input stream

Parameters:

file - a InputFile to read

filter - the filter to apply to row groups

Returns:

the metadata blocks in the footer

Throws:

IOException - if an error occurs while reading the file

open

@Deprecated
public static ParquetFileReader open(org.apache.hadoop.conf.Configuration conf,
                                                 org.apache.hadoop.fs.Path file)
                                          throws IOException

Deprecated. will be removed in 2.0.0; use open(InputFile)

Parameters:

conf - a configuration

file - a file path to open

Returns:

a parquet file reader

Throws:

IOException - if there is an error while opening the file

open

@Deprecated
public static ParquetFileReader open(org.apache.hadoop.conf.Configuration conf,
                                                 org.apache.hadoop.fs.Path file,
                                                 ParquetMetadataConverter.MetadataFilter filter)
                                          throws IOException

Deprecated. will be removed in 2.0.0; use open(InputFile,ParquetReadOptions)

Parameters:

conf - a configuration

file - a file path to open

filter - a metadata filter

Returns:

a parquet file reader

Throws:

IOException - if there is an error while opening the file

open

@Deprecated
public static ParquetFileReader open(org.apache.hadoop.conf.Configuration conf,
                                                 org.apache.hadoop.fs.Path file,
                                                 ParquetMetadata footer)
                                          throws IOException

Deprecated. will be removed in 2.0.0

Parameters:

conf - a configuration

file - a file path to open

footer - a footer for the file if already loaded

Returns:

a parquet file reader

Throws:

IOException - if there is an error while opening the file

open

public static ParquetFileReader open(InputFile file)
                              throws IOException

Open a file.

Parameters:

file - an input file

Returns:

an open ParquetFileReader

Throws:

IOException - if there is an error while opening the file

open

public static ParquetFileReader open(InputFile file,
                                     ParquetReadOptions options)
                              throws IOException

Open a file with options.

Parameters:

file - an input file

options - parquet read options

Returns:

an open ParquetFileReader

Throws:

IOException - if there is an error while opening the file

getFooter

public ParquetMetadata getFooter()

getFileMetaData

public FileMetaData getFileMetaData()

getRecordCount

public long getRecordCount()

getFilteredRecordCount

public long getFilteredRecordCount()

getPath

@Deprecated
public org.apache.hadoop.fs.Path getPath()

Deprecated. will be removed in 2.0.0; use getFile() instead

Returns:

the path for this file

getFile

public String getFile()

getRowGroups

public List<BlockMetaData> getRowGroups()

setRequestedSchema

public void setRequestedSchema(MessageType projection)

appendTo

public void appendTo(ParquetFileWriter writer)
              throws IOException

Throws:

IOException

readNextRowGroup

public PageReadStore readNextRowGroup()
                               throws IOException

Reads all the columns requested from the row group at the current file position.

Returns:

the PageReadStore which can provide PageReaders for each column.

Throws:

IOException - if an error occurs while reading

readNextFilteredRowGroup

public PageReadStore readNextFilteredRowGroup()
                                       throws IOException

Reads all the columns requested from the row group at the current file position. It may skip specific pages based on the column indexes according to the actual filter. As the rows are not aligned among the pages of the different columns row synchronization might be required. See the documentation of the class SynchronizingColumnReader for details.

Returns:

the PageReadStore which can provide PageReaders for each column

Throws:

IOException - if any I/O error occurs while reading

skipNextRowGroup

public boolean skipNextRowGroup()

getNextDictionaryReader

public DictionaryPageReadStore getNextDictionaryReader()

Returns a DictionaryPageReadStore for the row group that would be returned by calling readNextRowGroup() or skipped by calling skipNextRowGroup().

Returns:

a DictionaryPageReadStore for the next row group

getDictionaryReader

public org.apache.parquet.hadoop.DictionaryPageReader getDictionaryReader(BlockMetaData block)

readColumnIndex

@InterfaceAudience.Private
public ColumnIndex readColumnIndex(ColumnChunkMetaData column)
                                                       throws IOException

Parameters:

column - the column chunk which the column index is to be returned for

Returns:

the column index for the specified column chunk or null if there is no index

Throws:

IOException - if any I/O error occurs during reading the file

readOffsetIndex

@InterfaceAudience.Private
public OffsetIndex readOffsetIndex(ColumnChunkMetaData column)
                                                       throws IOException

Parameters:

column - the column chunk which the offset index is to be returned for

Returns:

the offset index for the specified column chunk or null if there is no index

Throws:

IOException - if any I/O error occurs during reading the file

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException