org.apache.hudi.integ.testsuite

Class HoodieContinousTestSuiteWriter

java.lang.Object

org.apache.hudi.integ.testsuite.HoodieTestSuiteWriter

org.apache.hudi.integ.testsuite.HoodieContinousTestSuiteWriter

All Implemented Interfaces:

Serializable

public class HoodieContinousTestSuiteWriter
extends HoodieTestSuiteWriter

Test suite Writer that assists in testing async table operations with Deltastreamer continuous mode. Sample command ./bin/spark-submit --packages org.apache.spark:spark-avro_2.11:2.4.4 \ --conf spark.task.cpus=1 --conf spark.executor.cores=1 \ --conf spark.task.maxFailures=100 \ --conf spark.memory.fraction=0.4 \ --conf spark.rdd.compress=true \ --conf spark.kryoserializer.buffer.max=2000m \ --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \ --conf spark.memory.storageFraction=0.1 \ --conf spark.shuffle.service.enabled=true \ --conf spark.sql.hive.convertMetastoreParquet=false \ --conf spark.driver.maxResultSize=12g \ --conf spark.executor.heartbeatInterval=120s \ --conf spark.network.timeout=600s \ --conf spark.yarn.max.executor.failures=10 \ --conf spark.sql.catalogImplementation=hive \ --class org.apache.hudi.integ.testsuite.HoodieTestSuiteJob /hudi-integ-test-bundle-0.12.0-SNAPSHOT.jar \ --source-ordering-field test_suite_source_ordering_field \ --use-deltastreamer \ --target-base-path /tmp/hudi/output \ --input-base-path /tmp/hudi/input \ --target-table table1 \ -props file:/tmp/test.properties \ --schemaprovider-class org.apache.hudi.integ.testsuite.schema.TestSuiteFileBasedSchemaProvider \ --source-class org.apache.hudi.utilities.sources.AvroDFSSource \ --input-file-size 125829120 \ --workload-yaml-path file:/tmp/simple-deltastreamer.yaml \ --workload-generator-classname org.apache.hudi.integ.testsuite.dag.WorkflowDagGenerator \ --table-type COPY_ON_WRITE \ --compact-scheduling-minshare 1 \ --clean-input \ --clean-output \ --continuous \ --test-continuous-mode \ --min-sync-interval-seconds 20

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.hudi.integ.testsuite.HoodieTestSuiteWriter

cfg, configuration, deltaStreamerWrapper, hoodieReadClient, lastCheckpoint, props, schema, sparkContext, VALID_DAG_NODES_TO_ALLOW_WRITE_CLIENT_IN_DELTASTREAMER_MODE, writeClient, writeConfig

Constructor Summary

Constructors

Constructor and Description
HoodieContinousTestSuiteWriter(org.apache.spark.api.java.JavaSparkContext jsc, Properties props, HoodieTestSuiteJob.HoodieTestSuiteConfig cfg, String schema)

Method Summary

Modifier and Type	Method and Description
org.apache.spark.api.java.JavaRDD<org.apache.hudi.client.WriteStatus>	bulkInsert(org.apache.hudi.common.util.Option<String> instantTime)
void	commit(org.apache.spark.api.java.JavaRDD<org.apache.hudi.client.WriteStatus> records, org.apache.spark.api.java.JavaRDD<DeltaWriteStats> generatedDataStats, org.apache.hudi.common.util.Option<String> instantTime)
void	commitCompaction(org.apache.spark.api.java.JavaRDD<org.apache.hudi.client.WriteStatus> records, org.apache.spark.api.java.JavaRDD<DeltaWriteStats> generatedDataStats, org.apache.hudi.common.util.Option<String> instantTime)
org.apache.spark.api.java.JavaRDD<org.apache.hudi.client.WriteStatus>	compact(org.apache.hudi.common.util.Option<String> instantTime)
org.apache.hudi.common.util.collection.Pair<SchemaProvider,org.apache.hudi.common.util.collection.Pair<String,org.apache.spark.api.java.JavaRDD<org.apache.hudi.common.model.HoodieRecord>>>	fetchSource()
org.apache.spark.rdd.RDD<org.apache.avro.generic.GenericRecord>	getNextBatch()
void	inlineClustering()
org.apache.spark.api.java.JavaRDD<org.apache.hudi.client.WriteStatus>	insert(org.apache.hudi.common.util.Option<String> instantTime)
org.apache.spark.api.java.JavaRDD<org.apache.hudi.client.WriteStatus>	insertOverwrite(org.apache.hudi.common.util.Option<String> instantTime)
org.apache.spark.api.java.JavaRDD<org.apache.hudi.client.WriteStatus>	insertOverwriteTable(org.apache.hudi.common.util.Option<String> instantTime)
org.apache.hudi.common.util.Option<String>	scheduleCompaction(org.apache.hudi.common.util.Option<Map<String,String>> previousCommitExtraMetadata)
void	shutdownResources()
org.apache.hudi.common.util.Option<String>	startCommit()
org.apache.spark.api.java.JavaRDD<org.apache.hudi.client.WriteStatus>	upsert(org.apache.hudi.common.util.Option<String> instantTime)

Methods inherited from class org.apache.hudi.integ.testsuite.HoodieTestSuiteWriter

getCfg, getConfiguration, getDeltaStreamerWrapper, getLastCheckpoint, getProps, getSchema, getSparkContext, getWriteClient, getWriteConfig

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HoodieContinousTestSuiteWriter

public HoodieContinousTestSuiteWriter(org.apache.spark.api.java.JavaSparkContext jsc,
                                      Properties props,
                                      HoodieTestSuiteJob.HoodieTestSuiteConfig cfg,
                                      String schema)
                               throws Exception

Throws:

Exception

Method Detail

shutdownResources

public void shutdownResources()

Specified by:

shutdownResources in class HoodieTestSuiteWriter

getNextBatch

public org.apache.spark.rdd.RDD<org.apache.avro.generic.GenericRecord> getNextBatch()
                                                                             throws Exception

Specified by:

getNextBatch in class HoodieTestSuiteWriter

Throws:

Exception

fetchSource

public org.apache.hudi.common.util.collection.Pair<SchemaProvider,org.apache.hudi.common.util.collection.Pair<String,org.apache.spark.api.java.JavaRDD<org.apache.hudi.common.model.HoodieRecord>>> fetchSource()
                                                                                                                                                                                                         throws Exception

Specified by:

fetchSource in class HoodieTestSuiteWriter

Throws:

Exception

startCommit

public org.apache.hudi.common.util.Option<String> startCommit()

Specified by:

startCommit in class HoodieTestSuiteWriter

upsert

public org.apache.spark.api.java.JavaRDD<org.apache.hudi.client.WriteStatus> upsert(org.apache.hudi.common.util.Option<String> instantTime)
                                                                             throws Exception

Specified by:

upsert in class HoodieTestSuiteWriter

Throws:

Exception

insert

public org.apache.spark.api.java.JavaRDD<org.apache.hudi.client.WriteStatus> insert(org.apache.hudi.common.util.Option<String> instantTime)
                                                                             throws Exception

Specified by:

insert in class HoodieTestSuiteWriter

Throws:

Exception

insertOverwrite

public org.apache.spark.api.java.JavaRDD<org.apache.hudi.client.WriteStatus> insertOverwrite(org.apache.hudi.common.util.Option<String> instantTime)
                                                                                      throws Exception

Specified by:

insertOverwrite in class HoodieTestSuiteWriter

Throws:

Exception

insertOverwriteTable

public org.apache.spark.api.java.JavaRDD<org.apache.hudi.client.WriteStatus> insertOverwriteTable(org.apache.hudi.common.util.Option<String> instantTime)
                                                                                           throws Exception

Specified by:

insertOverwriteTable in class HoodieTestSuiteWriter

Throws:

Exception

bulkInsert

public org.apache.spark.api.java.JavaRDD<org.apache.hudi.client.WriteStatus> bulkInsert(org.apache.hudi.common.util.Option<String> instantTime)
                                                                                 throws Exception

Specified by:

bulkInsert in class HoodieTestSuiteWriter

Throws:

Exception

compact

public org.apache.spark.api.java.JavaRDD<org.apache.hudi.client.WriteStatus> compact(org.apache.hudi.common.util.Option<String> instantTime)
                                                                              throws Exception

Specified by:

compact in class HoodieTestSuiteWriter

Throws:

Exception

inlineClustering

public void inlineClustering()

Specified by:

inlineClustering in class HoodieTestSuiteWriter

scheduleCompaction

public org.apache.hudi.common.util.Option<String> scheduleCompaction(org.apache.hudi.common.util.Option<Map<String,String>> previousCommitExtraMetadata)
                                                              throws Exception

Specified by:

scheduleCompaction in class HoodieTestSuiteWriter

Throws:

Exception

commit

public void commit(org.apache.spark.api.java.JavaRDD<org.apache.hudi.client.WriteStatus> records,
                   org.apache.spark.api.java.JavaRDD<DeltaWriteStats> generatedDataStats,
                   org.apache.hudi.common.util.Option<String> instantTime)

Specified by:

commit in class HoodieTestSuiteWriter

commitCompaction

public void commitCompaction(org.apache.spark.api.java.JavaRDD<org.apache.hudi.client.WriteStatus> records,
                             org.apache.spark.api.java.JavaRDD<DeltaWriteStats> generatedDataStats,
                             org.apache.hudi.common.util.Option<String> instantTime)
                      throws IOException

Specified by:

commitCompaction in class HoodieTestSuiteWriter

Throws:

IOException