org.apache.hudi.common.util

Class OrcUtils

java.lang.Object

org.apache.hudi.common.util.BaseFileUtils

org.apache.hudi.common.util.OrcUtils

public class OrcUtils
extends BaseFileUtils

Utility functions for ORC files.

Constructor Summary

Constructors

Constructor and Description
OrcUtils()

Method Summary

Modifier and Type	Method and Description
List<HoodieKey>	fetchHoodieKeys(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path filePath) Fetch HoodieKeys from the given ORC file.
List<HoodieKey>	fetchHoodieKeys(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path filePath, Option<BaseKeyGenerator> keyGeneratorOpt) Fetch HoodieKeys from the given data file.
Set<String>	filterRowKeys(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path filePath, Set<String> filter) Read the rowKey list matching the given filter, from the given ORC file.
HoodieFileFormat	getFormat()
ClosableIterator<HoodieKey>	getHoodieKeyIterator(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path filePath) Provides a closable iterator for reading the given ORC file.
ClosableIterator<HoodieKey>	getHoodieKeyIterator(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path filePath, Option<BaseKeyGenerator> keyGeneratorOpt) Provides a closable iterator for reading the given data file.
long	getRowCount(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path orcFilePath) Returns the number of records in the data file.
List<org.apache.avro.generic.GenericRecord>	readAvroRecords(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path filePath) NOTE: This literally reads the entire file contents, thus should be used with caution.
List<org.apache.avro.generic.GenericRecord>	readAvroRecords(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.fs.Path filePath, org.apache.avro.Schema avroSchema) NOTE: This literally reads the entire file contents, thus should be used with caution.
org.apache.avro.Schema	readAvroSchema(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path orcFilePath) Read the Avro schema of the data file.
Map<String,String>	readFooter(org.apache.hadoop.conf.Configuration conf, boolean required, org.apache.hadoop.fs.Path orcFilePath, String... footerNames) Read the footer data of the given data file.

Methods inherited from class org.apache.hudi.common.util.BaseFileUtils

getInstance, getInstance, getInstance, readBloomFilterFromMetadata, readMinMaxRecordKeys, readRowKeys

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

OrcUtils

public OrcUtils()

Method Detail

getHoodieKeyIterator

public ClosableIterator<HoodieKey> getHoodieKeyIterator(org.apache.hadoop.conf.Configuration configuration,
                                                        org.apache.hadoop.fs.Path filePath)

Provides a closable iterator for reading the given ORC file.

Specified by:

getHoodieKeyIterator in class BaseFileUtils

Parameters:

configuration - configuration to build fs object

filePath - The ORC file path

Returns:

ClosableIterator of HoodieKeys for reading the ORC file

fetchHoodieKeys

public List<HoodieKey> fetchHoodieKeys(org.apache.hadoop.conf.Configuration configuration,
                                       org.apache.hadoop.fs.Path filePath)

Fetch HoodieKeys from the given ORC file.

Specified by:

fetchHoodieKeys in class BaseFileUtils

Parameters:

filePath - The ORC file path.

configuration - configuration to build fs object

Returns:

List of HoodieKeys fetched from the ORC file

fetchHoodieKeys

public List<HoodieKey> fetchHoodieKeys(org.apache.hadoop.conf.Configuration configuration,
                                       org.apache.hadoop.fs.Path filePath,
                                       Option<BaseKeyGenerator> keyGeneratorOpt)

Description copied from class: BaseFileUtils

Fetch HoodieKeys from the given data file.

Specified by:

fetchHoodieKeys in class BaseFileUtils

Parameters:

configuration - configuration to build fs object

filePath - The data file path

keyGeneratorOpt - instance of KeyGenerator.

Returns:

List of HoodieKeys fetched from the data file

getHoodieKeyIterator

public ClosableIterator<HoodieKey> getHoodieKeyIterator(org.apache.hadoop.conf.Configuration configuration,
                                                        org.apache.hadoop.fs.Path filePath,
                                                        Option<BaseKeyGenerator> keyGeneratorOpt)

Description copied from class: BaseFileUtils

Provides a closable iterator for reading the given data file.

Specified by:

getHoodieKeyIterator in class BaseFileUtils

Parameters:

configuration - configuration to build fs object

filePath - The data file path

keyGeneratorOpt - instance of KeyGenerator.

Returns:

ClosableIterator of HoodieKeys for reading the file

readAvroRecords

public List<org.apache.avro.generic.GenericRecord> readAvroRecords(org.apache.hadoop.conf.Configuration configuration,
                                                                   org.apache.hadoop.fs.Path filePath)

NOTE: This literally reads the entire file contents, thus should be used with caution.

Specified by:

readAvroRecords in class BaseFileUtils

Parameters:

configuration - Configuration

filePath - The data file path

Returns:

A list of GenericRecord

readAvroRecords

public List<org.apache.avro.generic.GenericRecord> readAvroRecords(org.apache.hadoop.conf.Configuration configuration,
                                                                   org.apache.hadoop.fs.Path filePath,
                                                                   org.apache.avro.Schema avroSchema)

NOTE: This literally reads the entire file contents, thus should be used with caution.

Specified by:

readAvroRecords in class BaseFileUtils

Parameters:

configuration - Configuration

filePath - The data file path

Returns:

A list of GenericRecord

filterRowKeys

public Set<String> filterRowKeys(org.apache.hadoop.conf.Configuration conf,
                                 org.apache.hadoop.fs.Path filePath,
                                 Set<String> filter)
                          throws HoodieIOException

Read the rowKey list matching the given filter, from the given ORC file. If the filter is empty, then this will return all the rowkeys.

Specified by:

filterRowKeys in class BaseFileUtils

Parameters:

conf - configuration to build fs object.

filePath - The ORC file path.

filter - record keys filter

Returns:

Set Set of row keys matching candidateRecordKeys

Throws:

HoodieIOException

readFooter

public Map<String,String> readFooter(org.apache.hadoop.conf.Configuration conf,
                                     boolean required,
                                     org.apache.hadoop.fs.Path orcFilePath,
                                     String... footerNames)

Description copied from class: BaseFileUtils

Read the footer data of the given data file.

Specified by:

readFooter in class BaseFileUtils

Parameters:

conf - Configuration

required - require the footer data to be in data file

orcFilePath - The data file path

footerNames - The footer names to read

Returns:

A map where the key is the footer name and the value is the footer value

readAvroSchema

public org.apache.avro.Schema readAvroSchema(org.apache.hadoop.conf.Configuration conf,
                                             org.apache.hadoop.fs.Path orcFilePath)

Description copied from class: BaseFileUtils

Read the Avro schema of the data file.

Specified by:

readAvroSchema in class BaseFileUtils

Parameters:

conf - Configuration

orcFilePath - The data file path

Returns:

The Avro schema of the data file

getFormat

public HoodieFileFormat getFormat()

Specified by:

getFormat in class BaseFileUtils

Returns:

The subclass's HoodieFileFormat.

getRowCount

public long getRowCount(org.apache.hadoop.conf.Configuration conf,
                        org.apache.hadoop.fs.Path orcFilePath)

Description copied from class: BaseFileUtils

Returns the number of records in the data file.

Specified by:

getRowCount in class BaseFileUtils

Parameters:

conf - Configuration

orcFilePath - The data file path