org.apache.hudi.integ.testsuite

Class HoodieDeltaStreamerWrapper

java.lang.Object

org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer

org.apache.hudi.integ.testsuite.HoodieDeltaStreamerWrapper

All Implemented Interfaces:

Serializable

public class HoodieDeltaStreamerWrapper
extends HoodieDeltaStreamer

Extends the HoodieDeltaStreamer to expose certain operations helpful in running the Test Suite. This is done to achieve 2 things 1) Leverage some components of HoodieDeltaStreamer 2) Piggyback on the suite to test HoodieDeltaStreamer

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer

HoodieDeltaStreamer.Config, HoodieDeltaStreamer.DeltaSyncService

Field Summary

Fields inherited from class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer

cfg, CHECKPOINT_KEY, CHECKPOINT_RESET_KEY, DELTASYNC_POOL_NAME, deltaSyncService

Constructor Summary

Constructors

Constructor and Description
HoodieDeltaStreamerWrapper(HoodieDeltaStreamer.Config cfg, org.apache.spark.api.java.JavaSparkContext jssc)

Method Summary

Modifier and Type	Method and Description
org.apache.spark.api.java.JavaRDD<org.apache.hudi.client.WriteStatus>	bulkInsert()
org.apache.spark.api.java.JavaRDD<org.apache.hudi.client.WriteStatus>	compact()
org.apache.hudi.common.util.collection.Pair<SchemaProvider,org.apache.hudi.common.util.collection.Pair<String,org.apache.spark.api.java.JavaRDD<org.apache.hudi.common.model.HoodieRecord>>>	fetchSource()
org.apache.spark.api.java.JavaRDD<org.apache.hudi.client.WriteStatus>	insert()
org.apache.spark.api.java.JavaRDD<org.apache.hudi.client.WriteStatus>	insertOverwrite()
org.apache.spark.api.java.JavaRDD<org.apache.hudi.client.WriteStatus>	insertOverwriteTable()
void	scheduleCompact()
org.apache.spark.api.java.JavaRDD<org.apache.hudi.client.WriteStatus>	upsert(org.apache.hudi.common.model.WriteOperationType operation)

Methods inherited from class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer

getConfig, getConfig, getDeltaSyncService, main, shutdownGracefully, sync

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HoodieDeltaStreamerWrapper

public HoodieDeltaStreamerWrapper(HoodieDeltaStreamer.Config cfg,
                                  org.apache.spark.api.java.JavaSparkContext jssc)
                           throws Exception

Throws:

Exception

Method Detail

upsert

public org.apache.spark.api.java.JavaRDD<org.apache.hudi.client.WriteStatus> upsert(org.apache.hudi.common.model.WriteOperationType operation)
                                                                             throws Exception

Throws:

Exception

insert

public org.apache.spark.api.java.JavaRDD<org.apache.hudi.client.WriteStatus> insert()
                                                                             throws Exception

Throws:

Exception

bulkInsert

public org.apache.spark.api.java.JavaRDD<org.apache.hudi.client.WriteStatus> bulkInsert()
                                                                                 throws Exception

Throws:

Exception

insertOverwrite

public org.apache.spark.api.java.JavaRDD<org.apache.hudi.client.WriteStatus> insertOverwrite()
                                                                                      throws Exception

Throws:

Exception

insertOverwriteTable

public org.apache.spark.api.java.JavaRDD<org.apache.hudi.client.WriteStatus> insertOverwriteTable()
                                                                                           throws Exception

Throws:

Exception

scheduleCompact

public void scheduleCompact()
                     throws Exception

Throws:

Exception

compact

public org.apache.spark.api.java.JavaRDD<org.apache.hudi.client.WriteStatus> compact()
                                                                              throws Exception

Throws:

Exception

fetchSource

public org.apache.hudi.common.util.collection.Pair<SchemaProvider,org.apache.hudi.common.util.collection.Pair<String,org.apache.spark.api.java.JavaRDD<org.apache.hudi.common.model.HoodieRecord>>> fetchSource()
                                                                                                                                                                                                         throws Exception

Throws:

Exception