org.apache.hudi.common.util

Class OrcUtils

java.lang.Object

org.apache.hudi.common.util.FileFormatUtils

org.apache.hudi.common.util.OrcUtils

public class OrcUtils
extends FileFormatUtils

Utility functions for ORC files.

Constructor Summary

Constructors

Constructor and Description
OrcUtils()

Method Summary

Modifier and Type	Method and Description
List<HoodieKey>	fetchHoodieKeys(HoodieStorage storage, StoragePath filePath) Fetch HoodieKeys from the given ORC file.
List<HoodieKey>	fetchHoodieKeys(HoodieStorage storage, StoragePath filePath, Option<BaseKeyGenerator> keyGeneratorOpt)
Set<String>	filterRowKeys(HoodieStorage storage, StoragePath filePath, Set<String> filter) Read the rowKey list matching the given filter, from the given ORC file.
HoodieFileFormat	getFormat()
ClosableIterator<HoodieKey>	getHoodieKeyIterator(HoodieStorage storage, StoragePath filePath) Provides a closable iterator for reading the given ORC file.
ClosableIterator<HoodieKey>	getHoodieKeyIterator(HoodieStorage storage, StoragePath filePath, Option<BaseKeyGenerator> keyGeneratorOpt)
long	getRowCount(HoodieStorage storage, StoragePath filePath)
List<org.apache.avro.generic.GenericRecord>	readAvroRecords(HoodieStorage storage, StoragePath filePath) NOTE: This literally reads the entire file contents, thus should be used with caution.
List<org.apache.avro.generic.GenericRecord>	readAvroRecords(HoodieStorage storage, StoragePath filePath, org.apache.avro.Schema avroSchema) NOTE: This literally reads the entire file contents, thus should be used with caution.
org.apache.avro.Schema	readAvroSchema(HoodieStorage storage, StoragePath filePath)
List<HoodieColumnRangeMetadata<Comparable>>	readColumnStatsFromMetadata(HoodieStorage storage, StoragePath filePath, List<String> columnList)
Map<String,String>	readFooter(HoodieStorage storage, boolean required, StoragePath filePath, String... footerNames)
byte[]	serializeRecordsToLogBlock(HoodieStorage storage, List<HoodieRecord> records, org.apache.avro.Schema writerSchema, org.apache.avro.Schema readerSchema, String keyFieldName, Map<String,String> paramsMap)
void	writeMetaFile(HoodieStorage storage, StoragePath filePath, Properties props)

Methods inherited from class org.apache.hudi.common.util.FileFormatUtils

readBloomFilterFromMetadata, readMinMaxRecordKeys, readRowKeys

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

OrcUtils

public OrcUtils()

Method Detail

getHoodieKeyIterator

public ClosableIterator<HoodieKey> getHoodieKeyIterator(HoodieStorage storage,
                                                        StoragePath filePath)

Provides a closable iterator for reading the given ORC file.

Specified by:

getHoodieKeyIterator in class FileFormatUtils

Parameters:

storage - HoodieStorage instance.

filePath - The ORC file path

Returns:

ClosableIterator of HoodieKeys for reading the ORC file

fetchHoodieKeys

public List<HoodieKey> fetchHoodieKeys(HoodieStorage storage,
                                       StoragePath filePath)

Fetch HoodieKeys from the given ORC file.

Specified by:

fetchHoodieKeys in class FileFormatUtils

Parameters:

storage - HoodieStorage instance.

filePath - The ORC file path.

Returns:

List of HoodieKeys fetched from the ORC file

fetchHoodieKeys

public List<HoodieKey> fetchHoodieKeys(HoodieStorage storage,
                                       StoragePath filePath,
                                       Option<BaseKeyGenerator> keyGeneratorOpt)

Specified by:

fetchHoodieKeys in class FileFormatUtils

getHoodieKeyIterator

public ClosableIterator<HoodieKey> getHoodieKeyIterator(HoodieStorage storage,
                                                        StoragePath filePath,
                                                        Option<BaseKeyGenerator> keyGeneratorOpt)

Specified by:

getHoodieKeyIterator in class FileFormatUtils

readAvroRecords

public List<org.apache.avro.generic.GenericRecord> readAvroRecords(HoodieStorage storage,
                                                                   StoragePath filePath)

NOTE: This literally reads the entire file contents, thus should be used with caution.

Specified by:

readAvroRecords in class FileFormatUtils

readAvroRecords

public List<org.apache.avro.generic.GenericRecord> readAvroRecords(HoodieStorage storage,
                                                                   StoragePath filePath,
                                                                   org.apache.avro.Schema avroSchema)

NOTE: This literally reads the entire file contents, thus should be used with caution.

Specified by:

readAvroRecords in class FileFormatUtils

filterRowKeys

public Set<String> filterRowKeys(HoodieStorage storage,
                                 StoragePath filePath,
                                 Set<String> filter)
                          throws HoodieIOException

Read the rowKey list matching the given filter, from the given ORC file. If the filter is empty, then this will return all the rowkeys.

Specified by:

filterRowKeys in class FileFormatUtils

Parameters:

storage - HoodieStorage instance.

filePath - The ORC file path.

filter - record keys filter

Returns:

Set of row keys matching candidateRecordKeys

Throws:

HoodieIOException

readFooter

public Map<String,String> readFooter(HoodieStorage storage,
                                     boolean required,
                                     StoragePath filePath,
                                     String... footerNames)

Specified by:

readFooter in class FileFormatUtils

readAvroSchema

public org.apache.avro.Schema readAvroSchema(HoodieStorage storage,
                                             StoragePath filePath)

Specified by:

readAvroSchema in class FileFormatUtils

readColumnStatsFromMetadata

public List<HoodieColumnRangeMetadata<Comparable>> readColumnStatsFromMetadata(HoodieStorage storage,
                                                                               StoragePath filePath,
                                                                               List<String> columnList)

Specified by:

readColumnStatsFromMetadata in class FileFormatUtils

getFormat

public HoodieFileFormat getFormat()

Specified by:

getFormat in class FileFormatUtils

getRowCount

public long getRowCount(HoodieStorage storage,
                        StoragePath filePath)

Specified by:

getRowCount in class FileFormatUtils

writeMetaFile

public void writeMetaFile(HoodieStorage storage,
                          StoragePath filePath,
                          Properties props)
                   throws IOException

Specified by:

writeMetaFile in class FileFormatUtils

Throws:

IOException

serializeRecordsToLogBlock

public byte[] serializeRecordsToLogBlock(HoodieStorage storage,
                                         List<HoodieRecord> records,
                                         org.apache.avro.Schema writerSchema,
                                         org.apache.avro.Schema readerSchema,
                                         String keyFieldName,
                                         Map<String,String> paramsMap)
                                  throws IOException

Specified by:

serializeRecordsToLogBlock in class FileFormatUtils

Throws:

IOException