org.apache.hudi.io.storage

Class HoodieHFileReader<R extends org.apache.avro.generic.IndexedRecord>

java.lang.Object

org.apache.hudi.io.storage.HoodieHFileReader<R>

All Implemented Interfaces:

AutoCloseable, HoodieFileReader<R>

public class HoodieHFileReader<R extends org.apache.avro.generic.IndexedRecord>
extends Object
implements HoodieFileReader<R>

NOTE: PLEASE READ DOCS & COMMENTS CAREFULLY BEFORE MAKING CHANGES

HoodieFileReader implementation allowing to read from HFile.

Field Summary

Fields

Modifier and Type	Field and Description
static String	KEY_BLOOM_FILTER_META_BLOCK
static String	KEY_BLOOM_FILTER_TYPE_CODE
static String	KEY_FIELD_NAME
static String	KEY_MAX_RECORD
static String	KEY_MIN_RECORD
static String	SCHEMA_KEY

Constructor Summary

Constructors

Constructor and Description
HoodieHFileReader(org.apache.hadoop.conf.Configuration hadoopConf, org.apache.hadoop.fs.Path path, org.apache.hadoop.hbase.io.hfile.CacheConfig cacheConfig)
HoodieHFileReader(org.apache.hadoop.conf.Configuration hadoopConf, org.apache.hadoop.fs.Path path, org.apache.hadoop.hbase.io.hfile.CacheConfig cacheConfig, org.apache.hadoop.fs.FileSystem fs)
HoodieHFileReader(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path dummyPath, byte[] content, Option<org.apache.avro.Schema> schemaOpt)
HoodieHFileReader(org.apache.hadoop.fs.Path path, org.apache.hadoop.hbase.io.hfile.HFile.Reader reader, Option<org.apache.avro.Schema> schemaOpt)

Method Summary

Modifier and Type	Method and Description
void	close()
Set<String>	filterRowKeys(Set<String> candidateRowKeys) Filter keys by availability.
Option<R>	getRecordByKey(String key, org.apache.avro.Schema readerSchema)
ClosableIterator<R>	getRecordIterator(org.apache.avro.Schema readerSchema)
ClosableIterator<R>	getRecordsByKeyPrefixIterator(List<String> keyPrefixes, org.apache.avro.Schema readerSchema)
ClosableIterator<R>	getRecordsByKeysIterator(List<String> keys, org.apache.avro.Schema readerSchema)
org.apache.avro.Schema	getSchema()
long	getTotalRecords()
static <R extends org.apache.avro.generic.IndexedRecord> List<R>	readAllRecords(HoodieHFileReader<R> reader) NOTE: THIS SHOULD ONLY BE USED FOR TESTING, RECORDS ARE MATERIALIZED EAGERLY
BloomFilter	readBloomFilter()
String[]	readMinMaxRecordKeys()
static <R extends org.apache.avro.generic.IndexedRecord> List<R>	readRecords(HoodieHFileReader<R> reader, List<String> keys) NOTE: THIS SHOULD ONLY BE USED FOR TESTING, RECORDS ARE MATERIALIZED EAGERLY
static <R extends org.apache.avro.generic.IndexedRecord> List<R>	readRecords(HoodieHFileReader<R> reader, List<String> keys, org.apache.avro.Schema schema) NOTE: THIS SHOULD ONLY BE USED FOR TESTING, RECORDS ARE MATERIALIZED EAGERLY

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hudi.io.storage.HoodieFileReader

getRecordByKey, getRecordIterator, getRecordsByKeyPrefixIterator, getRecordsByKeysIterator

Field Detail

SCHEMA_KEY

public static final String SCHEMA_KEY

See Also:

Constant Field Values

KEY_BLOOM_FILTER_META_BLOCK

public static final String KEY_BLOOM_FILTER_META_BLOCK

See Also:

Constant Field Values

KEY_BLOOM_FILTER_TYPE_CODE

public static final String KEY_BLOOM_FILTER_TYPE_CODE

See Also:

Constant Field Values

KEY_FIELD_NAME

public static final String KEY_FIELD_NAME

See Also:

Constant Field Values

KEY_MIN_RECORD

public static final String KEY_MIN_RECORD

See Also:

Constant Field Values

KEY_MAX_RECORD

public static final String KEY_MAX_RECORD

See Also:

Constant Field Values

Constructor Detail

HoodieHFileReader

public HoodieHFileReader(org.apache.hadoop.conf.Configuration hadoopConf,
                         org.apache.hadoop.fs.Path path,
                         org.apache.hadoop.hbase.io.hfile.CacheConfig cacheConfig)
                  throws IOException

Throws:

IOException

HoodieHFileReader

public HoodieHFileReader(org.apache.hadoop.conf.Configuration hadoopConf,
                         org.apache.hadoop.fs.Path path,
                         org.apache.hadoop.hbase.io.hfile.CacheConfig cacheConfig,
                         org.apache.hadoop.fs.FileSystem fs)
                  throws IOException

Throws:

IOException

HoodieHFileReader

public HoodieHFileReader(org.apache.hadoop.fs.FileSystem fs,
                         org.apache.hadoop.fs.Path dummyPath,
                         byte[] content,
                         Option<org.apache.avro.Schema> schemaOpt)
                  throws IOException

Throws:

IOException

HoodieHFileReader

public HoodieHFileReader(org.apache.hadoop.fs.Path path,
                         org.apache.hadoop.hbase.io.hfile.HFile.Reader reader,
                         Option<org.apache.avro.Schema> schemaOpt)
                  throws IOException

Throws:

IOException

Method Detail

readMinMaxRecordKeys

public String[] readMinMaxRecordKeys()

Specified by:

readMinMaxRecordKeys in interface HoodieFileReader<R extends org.apache.avro.generic.IndexedRecord>

readBloomFilter

public BloomFilter readBloomFilter()

Specified by:

readBloomFilter in interface HoodieFileReader<R extends org.apache.avro.generic.IndexedRecord>

getSchema

public org.apache.avro.Schema getSchema()

Specified by:

getSchema in interface HoodieFileReader<R extends org.apache.avro.generic.IndexedRecord>

filterRowKeys

public Set<String> filterRowKeys(Set<String> candidateRowKeys)

Filter keys by availability.

Note: This method is performant when the caller passes in a sorted candidate keys.

Specified by:

filterRowKeys in interface HoodieFileReader<R extends org.apache.avro.generic.IndexedRecord>

Parameters:

candidateRowKeys - - Keys to check for the availability

Returns:

Subset of candidate keys that are available

getRecordByKey

public Option<R> getRecordByKey(String key,
                                org.apache.avro.Schema readerSchema)
                         throws IOException

Specified by:

getRecordByKey in interface HoodieFileReader<R extends org.apache.avro.generic.IndexedRecord>

Throws:

IOException

getRecordIterator

public ClosableIterator<R> getRecordIterator(org.apache.avro.Schema readerSchema)
                                      throws IOException

Specified by:

getRecordIterator in interface HoodieFileReader<R extends org.apache.avro.generic.IndexedRecord>

Throws:

IOException

getRecordsByKeysIterator

public ClosableIterator<R> getRecordsByKeysIterator(List<String> keys,
                                                    org.apache.avro.Schema readerSchema)
                                             throws IOException

Specified by:

getRecordsByKeysIterator in interface HoodieFileReader<R extends org.apache.avro.generic.IndexedRecord>

Throws:

IOException

getRecordsByKeyPrefixIterator

public ClosableIterator<R> getRecordsByKeyPrefixIterator(List<String> keyPrefixes,
                                                         org.apache.avro.Schema readerSchema)
                                                  throws IOException

Specified by:

getRecordsByKeyPrefixIterator in interface HoodieFileReader<R extends org.apache.avro.generic.IndexedRecord>

Throws:

IOException

getTotalRecords

public long getTotalRecords()

Specified by:

getTotalRecords in interface HoodieFileReader<R extends org.apache.avro.generic.IndexedRecord>

close

public void close()

Specified by:

close in interface AutoCloseable

Specified by:

close in interface HoodieFileReader<R extends org.apache.avro.generic.IndexedRecord>

readAllRecords

public static <R extends org.apache.avro.generic.IndexedRecord> List<R> readAllRecords(HoodieHFileReader<R> reader)
                                                                                throws IOException

NOTE: THIS SHOULD ONLY BE USED FOR TESTING, RECORDS ARE MATERIALIZED EAGERLY

Reads all the records with given schema

Throws:

IOException

readRecords

public static <R extends org.apache.avro.generic.IndexedRecord> List<R> readRecords(HoodieHFileReader<R> reader,
                                                                                    List<String> keys)
                                                                             throws IOException

NOTE: THIS SHOULD ONLY BE USED FOR TESTING, RECORDS ARE MATERIALIZED EAGERLY

Reads all the records with given schema and filtering keys.

Throws:

IOException

readRecords

public static <R extends org.apache.avro.generic.IndexedRecord> List<R> readRecords(HoodieHFileReader<R> reader,
                                                                                    List<String> keys,
                                                                                    org.apache.avro.Schema schema)
                                                                             throws IOException

NOTE: THIS SHOULD ONLY BE USED FOR TESTING, RECORDS ARE MATERIALIZED EAGERLY

Reads all the records with given schema and filtering keys.

Throws:

IOException