org.apache.hudi.integ.testsuite

Class SparkDataSourceContinuousIngestTool

java.lang.Object

org.apache.hudi.integ.testsuite.SparkDataSourceContinuousIngestTool

public class SparkDataSourceContinuousIngestTool
extends Object

Sample command ./bin/spark-submit --packages org.apache.spark:spark-avro_2.11:2.4.4 --driver-memory 4g --executor-memory 4g \ --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.sql.catalogImplementation=hive \ --class org.apache.hudi.integ.testsuite.SparkDSContinuousIngestTool \ ${HUDI_ROOT_DIR}/packaging/hudi-integ-test-bundle/target/hudi-integ-test-bundle-0.11.0-SNAPSHOT.jar \ --source-path file:${SOURCE_DIR}/spark_ds_continuous --checkpoint-file-path /tmp/hudi/checkpoint \ --base-path file:///tmp/hudi/tbl_path/ --props /tmp/hudi_props.out Contents of hudi.properties hoodie.insert.shuffle.parallelism=4 hoodie.upsert.shuffle.parallelism=4 hoodie.bulkinsert.shuffle.parallelism=4 hoodie.delete.shuffle.parallelism=4 hoodie.datasource.write.recordkey.field=VendorID hoodie.datasource.write.partitionpath.field=date_col hoodie.datasource.write.operation=upsert hoodie.datasource.write.precombine.field=tpep_pickup_datetime hoodie.metadata.enable=false hoodie.table.name=hudi_tbl

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SparkDataSourceContinuousIngestTool.Config

Constructor Summary

Constructors

Constructor and Description
SparkDataSourceContinuousIngestTool(org.apache.spark.api.java.JavaSparkContext jsc, SparkDataSourceContinuousIngestTool.Config cfg)

Method Summary

Modifier and Type	Method and Description
static void	main(String[] args)
void	run()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkDataSourceContinuousIngestTool

public SparkDataSourceContinuousIngestTool(org.apache.spark.api.java.JavaSparkContext jsc,
                                           SparkDataSourceContinuousIngestTool.Config cfg)

Method Detail

main

public static void main(String[] args)

run

public void run()