org.apache.hudi.table.action.commit

Class BaseMergeHelper<T extends HoodieRecordPayload,I,K,O>

java.lang.Object

org.apache.hudi.table.action.commit.BaseMergeHelper<T,I,K,O>

Direct Known Subclasses:

HoodieMergeHelper

public abstract class BaseMergeHelper<T extends HoodieRecordPayload,I,K,O>
extends Object

Helper to read records from previous version of base file and run Merge.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	BaseMergeHelper.UpdateHandler Consumer that dequeues records from queue and sends to Merge Handle.

Constructor Summary

Constructors

Constructor and Description
BaseMergeHelper()

Method Summary

Modifier and Type	Method and Description
protected Iterator<org.apache.avro.generic.GenericRecord>	getMergingIterator(HoodieTable<T,I,K,O> table, HoodieMergeHandle<T,I,K,O> mergeHandle, HoodieBaseFile baseFile, HoodieFileReader<org.apache.avro.generic.GenericRecord> reader, org.apache.avro.Schema readSchema, boolean externalSchemaTransformation) Create Parquet record iterator that provides a stitched view of record read from skeleton and bootstrap file.
abstract void	runMerge(HoodieTable<T,I,K,O> table, HoodieMergeHandle<T,I,K,O> upsertHandle) Read records from previous version of base file and merge.
protected org.apache.avro.generic.GenericRecord	transformRecordBasedOnNewSchema(org.apache.avro.generic.GenericDatumReader<org.apache.avro.generic.GenericRecord> gReader, org.apache.avro.generic.GenericDatumWriter<org.apache.avro.generic.GenericRecord> gWriter, ThreadLocal<org.apache.avro.io.BinaryEncoder> encoderCache, ThreadLocal<org.apache.avro.io.BinaryDecoder> decoderCache, org.apache.avro.generic.GenericRecord gRec)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BaseMergeHelper

public BaseMergeHelper()

Method Detail

runMerge

public abstract void runMerge(HoodieTable<T,I,K,O> table,
                              HoodieMergeHandle<T,I,K,O> upsertHandle)
                       throws IOException

Read records from previous version of base file and merge.

Parameters:

table - Hoodie Table

upsertHandle - Merge Handle

Throws:

IOException - in case of error

transformRecordBasedOnNewSchema

protected org.apache.avro.generic.GenericRecord transformRecordBasedOnNewSchema(org.apache.avro.generic.GenericDatumReader<org.apache.avro.generic.GenericRecord> gReader,
                                                                                org.apache.avro.generic.GenericDatumWriter<org.apache.avro.generic.GenericRecord> gWriter,
                                                                                ThreadLocal<org.apache.avro.io.BinaryEncoder> encoderCache,
                                                                                ThreadLocal<org.apache.avro.io.BinaryDecoder> decoderCache,
                                                                                org.apache.avro.generic.GenericRecord gRec)

getMergingIterator

protected Iterator<org.apache.avro.generic.GenericRecord> getMergingIterator(HoodieTable<T,I,K,O> table,
                                                                             HoodieMergeHandle<T,I,K,O> mergeHandle,
                                                                             HoodieBaseFile baseFile,
                                                                             HoodieFileReader<org.apache.avro.generic.GenericRecord> reader,
                                                                             org.apache.avro.Schema readSchema,
                                                                             boolean externalSchemaTransformation)
                                                                      throws IOException

Create Parquet record iterator that provides a stitched view of record read from skeleton and bootstrap file. Skeleton file is a representation of the bootstrap file inside the table, with just the bare bone fields needed for indexing, writing and other functionality.

Throws:

IOException