org.apache.hudi.common.util

Class BaseFileUtils

java.lang.Object

org.apache.hudi.common.util.BaseFileUtils

Direct Known Subclasses:

OrcUtils, ParquetUtils

public abstract class BaseFileUtils
extends Object

Constructor Summary

Constructors

Constructor and Description
BaseFileUtils()

Method Summary

Modifier and Type	Method and Description
abstract List<HoodieKey>	fetchHoodieKeys(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path filePath) Fetch HoodieKeys from the given data file.
abstract List<HoodieKey>	fetchHoodieKeys(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path filePath, Option<BaseKeyGenerator> keyGeneratorOpt) Fetch HoodieKeys from the given data file.
abstract Set<String>	filterRowKeys(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path filePath, Set<String> filter) Read the rowKey list matching the given filter, from the given data file.
abstract HoodieFileFormat	getFormat()
abstract ClosableIterator<HoodieKey>	getHoodieKeyIterator(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path filePath) Provides a closable iterator for reading the given data file.
abstract ClosableIterator<HoodieKey>	getHoodieKeyIterator(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path filePath, Option<BaseKeyGenerator> keyGeneratorOpt) Provides a closable iterator for reading the given data file.
static BaseFileUtils	getInstance(HoodieFileFormat fileFormat)
static BaseFileUtils	getInstance(HoodieTableMetaClient metaClient)
static BaseFileUtils	getInstance(String path)
abstract long	getRowCount(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path filePath) Returns the number of records in the data file.
abstract List<org.apache.avro.generic.GenericRecord>	readAvroRecords(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path filePath) Read the data file NOTE: This literally reads the entire file contents, thus should be used with caution.
abstract List<org.apache.avro.generic.GenericRecord>	readAvroRecords(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path filePath, org.apache.avro.Schema schema) Read the data file using the given schema NOTE: This literally reads the entire file contents, thus should be used with caution.
abstract org.apache.avro.Schema	readAvroSchema(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path filePath) Read the Avro schema of the data file.
BloomFilter	readBloomFilterFromMetadata(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path filePath) Read the bloom filter from the metadata of the given data file.
abstract Map<String,String>	readFooter(org.apache.hadoop.conf.Configuration configuration, boolean required, org.apache.hadoop.fs.Path filePath, String... footerNames) Read the footer data of the given data file.
String[]	readMinMaxRecordKeys(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path filePath) Read the min and max record key from the metadata of the given data file.
Set<String>	readRowKeys(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path filePath) Read the rowKey list from the given data file.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BaseFileUtils

public BaseFileUtils()

Method Detail

getInstance

public static BaseFileUtils getInstance(String path)

getInstance

public static BaseFileUtils getInstance(HoodieFileFormat fileFormat)

getInstance

public static BaseFileUtils getInstance(HoodieTableMetaClient metaClient)

readRowKeys

public Set<String> readRowKeys(org.apache.hadoop.conf.Configuration configuration,
                               org.apache.hadoop.fs.Path filePath)

Read the rowKey list from the given data file.

Parameters:

filePath - The data file path

configuration - configuration to build fs object

Returns:

Set Set of row keys

readBloomFilterFromMetadata

public BloomFilter readBloomFilterFromMetadata(org.apache.hadoop.conf.Configuration configuration,
                                               org.apache.hadoop.fs.Path filePath)

Read the bloom filter from the metadata of the given data file.

Parameters:

configuration - Configuration

filePath - The data file path

Returns:

a BloomFilter object

readMinMaxRecordKeys

public String[] readMinMaxRecordKeys(org.apache.hadoop.conf.Configuration configuration,
                                     org.apache.hadoop.fs.Path filePath)

Read the min and max record key from the metadata of the given data file.

Parameters:

configuration - Configuration

filePath - The data file path

Returns:

A array of two string where the first is min record key and the second is max record key

readAvroRecords

public abstract List<org.apache.avro.generic.GenericRecord> readAvroRecords(org.apache.hadoop.conf.Configuration configuration,
                                                                            org.apache.hadoop.fs.Path filePath)

Read the data file NOTE: This literally reads the entire file contents, thus should be used with caution.

Parameters:

configuration - Configuration

filePath - The data file path

Returns:

A list of GenericRecord

readAvroRecords

public abstract List<org.apache.avro.generic.GenericRecord> readAvroRecords(org.apache.hadoop.conf.Configuration configuration,
                                                                            org.apache.hadoop.fs.Path filePath,
                                                                            org.apache.avro.Schema schema)

Read the data file using the given schema NOTE: This literally reads the entire file contents, thus should be used with caution.

Parameters:

configuration - Configuration

filePath - The data file path

Returns:

A list of GenericRecord

readFooter

public abstract Map<String,String> readFooter(org.apache.hadoop.conf.Configuration configuration,
                                              boolean required,
                                              org.apache.hadoop.fs.Path filePath,
                                              String... footerNames)

Read the footer data of the given data file.

Parameters:

configuration - Configuration

required - require the footer data to be in data file

filePath - The data file path

footerNames - The footer names to read

Returns:

A map where the key is the footer name and the value is the footer value

getRowCount

public abstract long getRowCount(org.apache.hadoop.conf.Configuration configuration,
                                 org.apache.hadoop.fs.Path filePath)

Returns the number of records in the data file.

Parameters:

configuration - Configuration

filePath - The data file path

filterRowKeys

public abstract Set<String> filterRowKeys(org.apache.hadoop.conf.Configuration configuration,
                                          org.apache.hadoop.fs.Path filePath,
                                          Set<String> filter)

Read the rowKey list matching the given filter, from the given data file. If the filter is empty, then this will return all the row keys.

Parameters:

filePath - The data file path

configuration - configuration to build fs object

filter - record keys filter

Returns:

Set Set of row keys matching candidateRecordKeys

fetchHoodieKeys

public abstract List<HoodieKey> fetchHoodieKeys(org.apache.hadoop.conf.Configuration configuration,
                                                org.apache.hadoop.fs.Path filePath)

Fetch HoodieKeys from the given data file.

Parameters:

configuration - configuration to build fs object

filePath - The data file path

Returns:

List of HoodieKeys fetched from the data file

getHoodieKeyIterator

public abstract ClosableIterator<HoodieKey> getHoodieKeyIterator(org.apache.hadoop.conf.Configuration configuration,
                                                                 org.apache.hadoop.fs.Path filePath,
                                                                 Option<BaseKeyGenerator> keyGeneratorOpt)

Provides a closable iterator for reading the given data file.

Parameters:

configuration - configuration to build fs object

filePath - The data file path

keyGeneratorOpt - instance of KeyGenerator.

Returns:

ClosableIterator of HoodieKeys for reading the file

getHoodieKeyIterator

public abstract ClosableIterator<HoodieKey> getHoodieKeyIterator(org.apache.hadoop.conf.Configuration configuration,
                                                                 org.apache.hadoop.fs.Path filePath)

Provides a closable iterator for reading the given data file.

Parameters:

configuration - configuration to build fs object

filePath - The data file path

Returns:

ClosableIterator of HoodieKeys for reading the file

fetchHoodieKeys

public abstract List<HoodieKey> fetchHoodieKeys(org.apache.hadoop.conf.Configuration configuration,
                                                org.apache.hadoop.fs.Path filePath,
                                                Option<BaseKeyGenerator> keyGeneratorOpt)

Fetch HoodieKeys from the given data file.

Parameters:

configuration - configuration to build fs object

filePath - The data file path

keyGeneratorOpt - instance of KeyGenerator.

Returns:

List of HoodieKeys fetched from the data file

readAvroSchema

public abstract org.apache.avro.Schema readAvroSchema(org.apache.hadoop.conf.Configuration configuration,
                                                      org.apache.hadoop.fs.Path filePath)

Read the Avro schema of the data file.

Parameters:

configuration - Configuration

filePath - The data file path

Returns:

The Avro schema of the data file

getFormat

public abstract HoodieFileFormat getFormat()

Returns:

The subclass's HoodieFileFormat.