org.apache.hudi.common.util

Class ParquetUtils

java.lang.Object

org.apache.hudi.common.util.FileFormatUtils

org.apache.hudi.common.util.ParquetUtils

public class ParquetUtils
extends FileFormatUtils

Utility functions involving with parquet.

Constructor Summary

Constructors

Constructor and Description
ParquetUtils()

Method Summary

Modifier and Type	Method and Description
List<HoodieKey>	fetchHoodieKeys(HoodieStorage storage, StoragePath filePath) Fetch HoodieKeys from the given parquet file.
List<HoodieKey>	fetchHoodieKeys(HoodieStorage storage, StoragePath filePath, Option<BaseKeyGenerator> keyGeneratorOpt) Fetch HoodieKeys from the given parquet file.
Set<String>	filterRowKeys(HoodieStorage storage, StoragePath filePath, Set<String> filter) Read the rowKey list matching the given filter, from the given parquet file.
static org.apache.parquet.hadoop.metadata.CompressionCodecName	getCompressionCodecName(String codecName)
HoodieFileFormat	getFormat()
ClosableIterator<HoodieKey>	getHoodieKeyIterator(HoodieStorage storage, StoragePath filePath)
ClosableIterator<HoodieKey>	getHoodieKeyIterator(HoodieStorage storage, StoragePath filePath, Option<BaseKeyGenerator> keyGeneratorOpt) Returns a closable iterator for reading the given parquet file.
long	getRowCount(HoodieStorage storage, StoragePath filePath) Returns the number of records in the parquet file.
List<org.apache.avro.generic.GenericRecord>	readAvroRecords(HoodieStorage storage, StoragePath filePath) NOTE: This literally reads the entire file contents, thus should be used with caution.
List<org.apache.avro.generic.GenericRecord>	readAvroRecords(HoodieStorage storage, StoragePath filePath, org.apache.avro.Schema schema)
org.apache.avro.Schema	readAvroSchema(HoodieStorage storage, StoragePath filePath)
List<HoodieColumnRangeMetadata<Comparable>>	readColumnStatsFromMetadata(HoodieStorage storage, StoragePath filePath, List<String> columnList)
Map<String,String>	readFooter(HoodieStorage storage, boolean required, StoragePath filePath, String... footerNames)
static org.apache.parquet.hadoop.metadata.ParquetMetadata	readMetadata(HoodieStorage storage, StoragePath parquetFilePath)
org.apache.parquet.schema.MessageType	readSchema(HoodieStorage storage, StoragePath parquetFilePath) Get the schema of the given parquet file.
byte[]	serializeRecordsToLogBlock(HoodieStorage storage, List<HoodieRecord> records, org.apache.avro.Schema writerSchema, org.apache.avro.Schema readerSchema, String keyFieldName, Map<String,String> paramsMap)
void	writeMetaFile(HoodieStorage storage, StoragePath filePath, Properties props)

Methods inherited from class org.apache.hudi.common.util.FileFormatUtils

readBloomFilterFromMetadata, readMinMaxRecordKeys, readRowKeys

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ParquetUtils

public ParquetUtils()

Method Detail

filterRowKeys

public Set<String> filterRowKeys(HoodieStorage storage,
                                 StoragePath filePath,
                                 Set<String> filter)

Read the rowKey list matching the given filter, from the given parquet file. If the filter is empty, then this will return all the rowkeys.

Specified by:

filterRowKeys in class FileFormatUtils

Parameters:

storage - HoodieStorage instance.

filePath - The parquet file path.

filter - record keys filter

Returns:

Set of row keys matching candidateRecordKeys

readMetadata

public static org.apache.parquet.hadoop.metadata.ParquetMetadata readMetadata(HoodieStorage storage,
                                                                              StoragePath parquetFilePath)

getCompressionCodecName

public static org.apache.parquet.hadoop.metadata.CompressionCodecName getCompressionCodecName(String codecName)

Parameters:

codecName - codec name in String.

Returns:

CompressionCodecName Enum.

fetchHoodieKeys

public List<HoodieKey> fetchHoodieKeys(HoodieStorage storage,
                                       StoragePath filePath)

Fetch HoodieKeys from the given parquet file.

Specified by:

fetchHoodieKeys in class FileFormatUtils

Parameters:

storage - HoodieStorage instance.

filePath - The parquet file path.

Returns:

List of HoodieKeys fetched from the parquet file

getHoodieKeyIterator

public ClosableIterator<HoodieKey> getHoodieKeyIterator(HoodieStorage storage,
                                                        StoragePath filePath)

Specified by:

getHoodieKeyIterator in class FileFormatUtils

getHoodieKeyIterator

public ClosableIterator<HoodieKey> getHoodieKeyIterator(HoodieStorage storage,
                                                        StoragePath filePath,
                                                        Option<BaseKeyGenerator> keyGeneratorOpt)

Returns a closable iterator for reading the given parquet file.

Specified by:

getHoodieKeyIterator in class FileFormatUtils

Parameters:

storage - HoodieStorage instance.

filePath - The parquet file path

keyGeneratorOpt - instance of KeyGenerator

Returns:

ClosableIterator of HoodieKeys for reading the parquet file

fetchHoodieKeys

public List<HoodieKey> fetchHoodieKeys(HoodieStorage storage,
                                       StoragePath filePath,
                                       Option<BaseKeyGenerator> keyGeneratorOpt)

Fetch HoodieKeys from the given parquet file.

Specified by:

fetchHoodieKeys in class FileFormatUtils

Parameters:

storage - HoodieStorage instance.

filePath - The parquet file path.

keyGeneratorOpt - instance of KeyGenerator.

Returns:

List of HoodieKeys fetched from the parquet file

readSchema

public org.apache.parquet.schema.MessageType readSchema(HoodieStorage storage,
                                                        StoragePath parquetFilePath)

Get the schema of the given parquet file.

readFooter

public Map<String,String> readFooter(HoodieStorage storage,
                                     boolean required,
                                     StoragePath filePath,
                                     String... footerNames)

Specified by:

readFooter in class FileFormatUtils

readAvroSchema

public org.apache.avro.Schema readAvroSchema(HoodieStorage storage,
                                             StoragePath filePath)

Specified by:

readAvroSchema in class FileFormatUtils

readColumnStatsFromMetadata

public List<HoodieColumnRangeMetadata<Comparable>> readColumnStatsFromMetadata(HoodieStorage storage,
                                                                               StoragePath filePath,
                                                                               List<String> columnList)

Specified by:

readColumnStatsFromMetadata in class FileFormatUtils

getFormat

public HoodieFileFormat getFormat()

Specified by:

getFormat in class FileFormatUtils

readAvroRecords

public List<org.apache.avro.generic.GenericRecord> readAvroRecords(HoodieStorage storage,
                                                                   StoragePath filePath)

NOTE: This literally reads the entire file contents, thus should be used with caution.

Specified by:

readAvroRecords in class FileFormatUtils

readAvroRecords

public List<org.apache.avro.generic.GenericRecord> readAvroRecords(HoodieStorage storage,
                                                                   StoragePath filePath,
                                                                   org.apache.avro.Schema schema)

Specified by:

readAvroRecords in class FileFormatUtils

getRowCount

public long getRowCount(HoodieStorage storage,
                        StoragePath filePath)

Returns the number of records in the parquet file.

Specified by:

getRowCount in class FileFormatUtils

Parameters:

storage - HoodieStorage instance.

filePath - path of the file

writeMetaFile

public void writeMetaFile(HoodieStorage storage,
                          StoragePath filePath,
                          Properties props)
                   throws IOException

Specified by:

writeMetaFile in class FileFormatUtils

Throws:

IOException

serializeRecordsToLogBlock

public byte[] serializeRecordsToLogBlock(HoodieStorage storage,
                                         List<HoodieRecord> records,
                                         org.apache.avro.Schema writerSchema,
                                         org.apache.avro.Schema readerSchema,
                                         String keyFieldName,
                                         Map<String,String> paramsMap)
                                  throws IOException

Specified by:

serializeRecordsToLogBlock in class FileFormatUtils

Throws:

IOException