org.apache.hudi.common.util

Class ParquetUtils

java.lang.Object

org.apache.hudi.common.util.BaseFileUtils

org.apache.hudi.common.util.ParquetUtils

public class ParquetUtils
extends BaseFileUtils

Utility functions involving with parquet.

Constructor Summary

Constructors

Constructor and Description
ParquetUtils()

Method Summary

Modifier and Type	Method and Description
List<HoodieKey>	fetchHoodieKeys(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path filePath) Fetch HoodieKeys from the given parquet file.
List<HoodieKey>	fetchHoodieKeys(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path filePath, Option<BaseKeyGenerator> keyGeneratorOpt) Fetch HoodieKeys from the given parquet file.
Set<String>	filterRowKeys(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path filePath, Set<String> filter) Read the rowKey list matching the given filter, from the given parquet file.
HoodieFileFormat	getFormat()
ClosableIterator<HoodieKey>	getHoodieKeyIterator(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path filePath) Provides a closable iterator for reading the given data file.
ClosableIterator<HoodieKey>	getHoodieKeyIterator(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path filePath, Option<BaseKeyGenerator> keyGeneratorOpt) Returns a closable iterator for reading the given parquet file.
long	getRowCount(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path parquetFilePath) Returns the number of records in the parquet file.
List<org.apache.avro.generic.GenericRecord>	readAvroRecords(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path filePath) NOTE: This literally reads the entire file contents, thus should be used with caution.
List<org.apache.avro.generic.GenericRecord>	readAvroRecords(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path filePath, org.apache.avro.Schema schema) Read the data file using the given schema NOTE: This literally reads the entire file contents, thus should be used with caution.
org.apache.avro.Schema	readAvroSchema(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path parquetFilePath) Read the Avro schema of the data file.
Map<String,String>	readFooter(org.apache.hadoop.conf.Configuration configuration, boolean required, org.apache.hadoop.fs.Path parquetFilePath, String... footerNames) Read the footer data of the given data file.
static org.apache.parquet.hadoop.metadata.ParquetMetadata	readMetadata(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path parquetFilePath)
List<HoodieColumnRangeMetadata<Comparable>>	readRangeFromParquetMetadata(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path parquetFilePath, List<String> cols) Parse min/max statistics stored in parquet footers for all columns.
org.apache.parquet.schema.MessageType	readSchema(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path parquetFilePath) Get the schema of the given parquet file.

Methods inherited from class org.apache.hudi.common.util.BaseFileUtils

getInstance, getInstance, getInstance, readBloomFilterFromMetadata, readMinMaxRecordKeys, readRowKeys

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ParquetUtils

public ParquetUtils()

Method Detail

filterRowKeys

public Set<String> filterRowKeys(org.apache.hadoop.conf.Configuration configuration,
                                 org.apache.hadoop.fs.Path filePath,
                                 Set<String> filter)

Read the rowKey list matching the given filter, from the given parquet file. If the filter is empty, then this will return all the rowkeys.

Specified by:

filterRowKeys in class BaseFileUtils

Parameters:

filePath - The parquet file path.

configuration - configuration to build fs object

filter - record keys filter

Returns:

Set Set of row keys matching candidateRecordKeys

readMetadata

public static org.apache.parquet.hadoop.metadata.ParquetMetadata readMetadata(org.apache.hadoop.conf.Configuration conf,
                                                                              org.apache.hadoop.fs.Path parquetFilePath)

fetchHoodieKeys

public List<HoodieKey> fetchHoodieKeys(org.apache.hadoop.conf.Configuration configuration,
                                       org.apache.hadoop.fs.Path filePath)

Fetch HoodieKeys from the given parquet file.

Specified by:

fetchHoodieKeys in class BaseFileUtils

Parameters:

filePath - The parquet file path.

configuration - configuration to build fs object

Returns:

List of HoodieKeys fetched from the parquet file

getHoodieKeyIterator

public ClosableIterator<HoodieKey> getHoodieKeyIterator(org.apache.hadoop.conf.Configuration configuration,
                                                        org.apache.hadoop.fs.Path filePath)

Description copied from class: BaseFileUtils

Provides a closable iterator for reading the given data file.

Specified by:

getHoodieKeyIterator in class BaseFileUtils

Parameters:

configuration - configuration to build fs object

filePath - The data file path

Returns:

ClosableIterator of HoodieKeys for reading the file

getHoodieKeyIterator

public ClosableIterator<HoodieKey> getHoodieKeyIterator(org.apache.hadoop.conf.Configuration configuration,
                                                        org.apache.hadoop.fs.Path filePath,
                                                        Option<BaseKeyGenerator> keyGeneratorOpt)

Returns a closable iterator for reading the given parquet file.

Specified by:

getHoodieKeyIterator in class BaseFileUtils

Parameters:

configuration - configuration to build fs object

filePath - The parquet file path

keyGeneratorOpt - instance of KeyGenerator

Returns:

ClosableIterator of HoodieKeys for reading the parquet file

fetchHoodieKeys

public List<HoodieKey> fetchHoodieKeys(org.apache.hadoop.conf.Configuration configuration,
                                       org.apache.hadoop.fs.Path filePath,
                                       Option<BaseKeyGenerator> keyGeneratorOpt)

Fetch HoodieKeys from the given parquet file.

Specified by:

fetchHoodieKeys in class BaseFileUtils

Parameters:

configuration - configuration to build fs object

filePath - The parquet file path.

keyGeneratorOpt - instance of KeyGenerator.

Returns:

List of HoodieKeys fetched from the parquet file

readSchema

public org.apache.parquet.schema.MessageType readSchema(org.apache.hadoop.conf.Configuration configuration,
                                                        org.apache.hadoop.fs.Path parquetFilePath)

Get the schema of the given parquet file.

readFooter

public Map<String,String> readFooter(org.apache.hadoop.conf.Configuration configuration,
                                     boolean required,
                                     org.apache.hadoop.fs.Path parquetFilePath,
                                     String... footerNames)

Description copied from class: BaseFileUtils

Read the footer data of the given data file.

Specified by:

readFooter in class BaseFileUtils

Parameters:

configuration - Configuration

required - require the footer data to be in data file

parquetFilePath - The data file path

footerNames - The footer names to read

Returns:

A map where the key is the footer name and the value is the footer value

readAvroSchema

public org.apache.avro.Schema readAvroSchema(org.apache.hadoop.conf.Configuration configuration,
                                             org.apache.hadoop.fs.Path parquetFilePath)

Description copied from class: BaseFileUtils

Read the Avro schema of the data file.

Specified by:

readAvroSchema in class BaseFileUtils

Parameters:

configuration - Configuration

parquetFilePath - The data file path

Returns:

The Avro schema of the data file

getFormat

public HoodieFileFormat getFormat()

Specified by:

getFormat in class BaseFileUtils

Returns:

The subclass's HoodieFileFormat.

readAvroRecords

public List<org.apache.avro.generic.GenericRecord> readAvroRecords(org.apache.hadoop.conf.Configuration configuration,
                                                                   org.apache.hadoop.fs.Path filePath)

NOTE: This literally reads the entire file contents, thus should be used with caution.

Specified by:

readAvroRecords in class BaseFileUtils

Parameters:

configuration - Configuration

filePath - The data file path

Returns:

A list of GenericRecord

readAvroRecords

public List<org.apache.avro.generic.GenericRecord> readAvroRecords(org.apache.hadoop.conf.Configuration configuration,
                                                                   org.apache.hadoop.fs.Path filePath,
                                                                   org.apache.avro.Schema schema)

Description copied from class: BaseFileUtils

Read the data file using the given schema NOTE: This literally reads the entire file contents, thus should be used with caution.

Specified by:

readAvroRecords in class BaseFileUtils

Parameters:

configuration - Configuration

filePath - The data file path

Returns:

A list of GenericRecord

getRowCount

public long getRowCount(org.apache.hadoop.conf.Configuration conf,
                        org.apache.hadoop.fs.Path parquetFilePath)

Returns the number of records in the parquet file.

Specified by:

getRowCount in class BaseFileUtils

Parameters:

conf - Configuration

parquetFilePath - path of the file

readRangeFromParquetMetadata

public List<HoodieColumnRangeMetadata<Comparable>> readRangeFromParquetMetadata(@Nonnull
                                                                                org.apache.hadoop.conf.Configuration conf,
                                                                                @Nonnull
                                                                                org.apache.hadoop.fs.Path parquetFilePath,
                                                                                @Nonnull
                                                                                List<String> cols)

Parse min/max statistics stored in parquet footers for all columns.