org.apache.hudi.common.util

Class FileFormatUtils

java.lang.Object

org.apache.hudi.common.util.FileFormatUtils

public abstract class FileFormatUtils
extends Object

Utils for file format used in Hudi.

Constructor Summary

Constructors

Constructor and Description
FileFormatUtils()

Method Summary

Modifier and Type	Method and Description
abstract List<HoodieKey>	fetchHoodieKeys(HoodieStorage storage, StoragePath filePath) Fetch HoodieKeys from the given data file.
abstract List<HoodieKey>	fetchHoodieKeys(HoodieStorage storage, StoragePath filePath, Option<BaseKeyGenerator> keyGeneratorOpt) Fetch HoodieKeys from the given data file.
abstract Set<String>	filterRowKeys(HoodieStorage storage, StoragePath filePath, Set<String> filter) Read the rowKey list matching the given filter, from the given data file.
abstract HoodieFileFormat	getFormat()
abstract ClosableIterator<HoodieKey>	getHoodieKeyIterator(HoodieStorage storage, StoragePath filePath) Provides a closable iterator for reading the given data file.
abstract ClosableIterator<HoodieKey>	getHoodieKeyIterator(HoodieStorage storage, StoragePath filePath, Option<BaseKeyGenerator> keyGeneratorOpt) Provides a closable iterator for reading the given data file.
abstract long	getRowCount(HoodieStorage storage, StoragePath filePath) Returns the number of records in the data file.
abstract List<org.apache.avro.generic.GenericRecord>	readAvroRecords(HoodieStorage storage, StoragePath filePath) Read the data file NOTE: This literally reads the entire file contents, thus should be used with caution.
abstract List<org.apache.avro.generic.GenericRecord>	readAvroRecords(HoodieStorage storage, StoragePath filePath, org.apache.avro.Schema schema) Read the data file using the given schema NOTE: This literally reads the entire file contents, thus should be used with caution.
abstract org.apache.avro.Schema	readAvroSchema(HoodieStorage storage, StoragePath filePath) Read the Avro schema of the data file.
BloomFilter	readBloomFilterFromMetadata(HoodieStorage storage, StoragePath filePath) Read the bloom filter from the metadata of the given data file.
abstract List<HoodieColumnRangeMetadata<Comparable>>	readColumnStatsFromMetadata(HoodieStorage storage, StoragePath filePath, List<String> columnList) Reads column statistics stored in the metadata.
abstract Map<String,String>	readFooter(HoodieStorage storage, boolean required, StoragePath filePath, String... footerNames) Read the footer data of the given data file.
String[]	readMinMaxRecordKeys(HoodieStorage storage, StoragePath filePath) Read the min and max record key from the metadata of the given data file.
Set<String>	readRowKeys(HoodieStorage storage, StoragePath filePath) Read the rowKey list from the given data file.
abstract byte[]	serializeRecordsToLogBlock(HoodieStorage storage, List<HoodieRecord> records, org.apache.avro.Schema writerSchema, org.apache.avro.Schema readerSchema, String keyFieldName, Map<String,String> paramsMap) Serializes Hudi records to the log block.
abstract void	writeMetaFile(HoodieStorage storage, StoragePath filePath, Properties props) Writes properties to the meta file.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FileFormatUtils

public FileFormatUtils()

Method Detail

readRowKeys

public Set<String> readRowKeys(HoodieStorage storage,
                               StoragePath filePath)

Read the rowKey list from the given data file.

Parameters:

storage - HoodieStorage instance.

filePath - the data file path.

Returns:

set of row keys

readBloomFilterFromMetadata

public BloomFilter readBloomFilterFromMetadata(HoodieStorage storage,
                                               StoragePath filePath)

Read the bloom filter from the metadata of the given data file.

Parameters:

storage - HoodieStorage instance.

filePath - the data file path.

Returns:

a BloomFilter object.

readMinMaxRecordKeys

public String[] readMinMaxRecordKeys(HoodieStorage storage,
                                     StoragePath filePath)

Read the min and max record key from the metadata of the given data file.

Parameters:

storage - HoodieStorage instance.

filePath - the data file path.

Returns:

an array of two string where the first is min record key and the second is max record key.

readAvroRecords

public abstract List<org.apache.avro.generic.GenericRecord> readAvroRecords(HoodieStorage storage,
                                                                            StoragePath filePath)

Read the data file NOTE: This literally reads the entire file contents, thus should be used with caution.

Parameters:

storage - HoodieStorage instance.

filePath - the data file path.

Returns:

a list of GenericRecord.

readAvroRecords

public abstract List<org.apache.avro.generic.GenericRecord> readAvroRecords(HoodieStorage storage,
                                                                            StoragePath filePath,
                                                                            org.apache.avro.Schema schema)

Read the data file using the given schema NOTE: This literally reads the entire file contents, thus should be used with caution.

Parameters:

storage - HoodieStorage instance.

filePath - the data file path.

Returns:

a list of GenericRecord.

readFooter

public abstract Map<String,String> readFooter(HoodieStorage storage,
                                              boolean required,
                                              StoragePath filePath,
                                              String... footerNames)

Read the footer data of the given data file.

Parameters:

storage - HoodieStorage instance.

required - require the footer data to be in data file.

filePath - the data file path.

footerNames - the footer names to read.

Returns:

a map where the key is the footer name and the value is the footer value.

getRowCount

public abstract long getRowCount(HoodieStorage storage,
                                 StoragePath filePath)

Returns the number of records in the data file.

Parameters:

storage - HoodieStorage instance.

filePath - the data file path.

filterRowKeys

public abstract Set<String> filterRowKeys(HoodieStorage storage,
                                          StoragePath filePath,
                                          Set<String> filter)

Read the rowKey list matching the given filter, from the given data file. If the filter is empty, then this will return all the row keys.

Parameters:

storage - HoodieStorage instance.

filePath - the data file path.

filter - record keys filter.

Returns:

set of row keys matching candidateRecordKeys.

fetchHoodieKeys

public abstract List<HoodieKey> fetchHoodieKeys(HoodieStorage storage,
                                                StoragePath filePath)

Fetch HoodieKeys from the given data file.

Parameters:

storage - HoodieStorage instance.

filePath - the data file path.

Returns:

List of HoodieKeys fetched from the data file.

getHoodieKeyIterator

public abstract ClosableIterator<HoodieKey> getHoodieKeyIterator(HoodieStorage storage,
                                                                 StoragePath filePath,
                                                                 Option<BaseKeyGenerator> keyGeneratorOpt)

Provides a closable iterator for reading the given data file.

Parameters:

storage - HoodieStorage instance.

filePath - the data file path.

keyGeneratorOpt - instance of KeyGenerator.

Returns:

ClosableIterator of HoodieKeys for reading the file.

getHoodieKeyIterator

public abstract ClosableIterator<HoodieKey> getHoodieKeyIterator(HoodieStorage storage,
                                                                 StoragePath filePath)

Provides a closable iterator for reading the given data file.

Parameters:

storage - HoodieStorage instance.

filePath - the data file path.

Returns:

ClosableIterator of HoodieKeys for reading the file.

fetchHoodieKeys

public abstract List<HoodieKey> fetchHoodieKeys(HoodieStorage storage,
                                                StoragePath filePath,
                                                Option<BaseKeyGenerator> keyGeneratorOpt)

Fetch HoodieKeys from the given data file.

Parameters:

storage - HoodieStorage instance.

filePath - the data file path.

keyGeneratorOpt - instance of KeyGenerator.

Returns:

List ofHoodieKeys fetched from the data file.

readAvroSchema

public abstract org.apache.avro.Schema readAvroSchema(HoodieStorage storage,
                                                      StoragePath filePath)

Read the Avro schema of the data file.

Parameters:

storage - HoodieStorage instance.

filePath - the data file path.

Returns:

the Avro schema of the data file.

readColumnStatsFromMetadata

public abstract List<HoodieColumnRangeMetadata<Comparable>> readColumnStatsFromMetadata(HoodieStorage storage,
                                                                                        StoragePath filePath,
                                                                                        List<String> columnList)

Reads column statistics stored in the metadata.

Parameters:

storage - HoodieStorage instance.

filePath - the data file path.

columnList - List of columns to get column statistics.

Returns:

List of HoodieColumnRangeMetadata.

getFormat

public abstract HoodieFileFormat getFormat()

Returns:

The subclass's HoodieFileFormat.

writeMetaFile

public abstract void writeMetaFile(HoodieStorage storage,
                                   StoragePath filePath,
                                   Properties props)
                            throws IOException

Writes properties to the meta file.

Parameters:

storage - HoodieStorage instance.

filePath - file path to write to.

props - properties to write.

Throws:

IOException - upon write error.

serializeRecordsToLogBlock

public abstract byte[] serializeRecordsToLogBlock(HoodieStorage storage,
                                                  List<HoodieRecord> records,
                                                  org.apache.avro.Schema writerSchema,
                                                  org.apache.avro.Schema readerSchema,
                                                  String keyFieldName,
                                                  Map<String,String> paramsMap)
                                           throws IOException

Serializes Hudi records to the log block.

Parameters:

storage - HoodieStorage instance.

records - a list of HoodieRecord.

writerSchema - writer schema string from the log block header.

readerSchema -

keyFieldName -

paramsMap - additional params for serialization.

Returns:

byte array after serialization.

Throws:

IOException - upon serialization error.