org.apache.hudi.utilities.deltastreamer

Class HoodieDeltaStreamer.DeltaSyncService

java.lang.Object

org.apache.hudi.utilities.deltastreamer.AbstractDeltaStreamerService

org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.DeltaSyncService

All Implemented Interfaces:

Serializable

Enclosing class:

HoodieDeltaStreamer

public static class HoodieDeltaStreamer.DeltaSyncService
extends AbstractDeltaStreamerService

Syncs data either in single-run or in continuous mode.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.hudi.utilities.deltastreamer.AbstractDeltaStreamerService

log

Constructor Summary

Constructors

Constructor and Description
DeltaSyncService(HoodieDeltaStreamer.Config cfg, org.apache.spark.api.java.JavaSparkContext jssc, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.hive.conf.HiveConf hiveConf)

Method Summary

Modifier and Type	Method and Description
void	close() Close all resources
HoodieDeltaStreamer.AsyncCompactService	getAsyncCompactService()
DeltaSync	getDeltaSync()
org.apache.spark.api.java.JavaSparkContext	getJavaSparkContext()
TypedProperties	getProps()
SchemaProvider	getSchemaProvider()
org.apache.spark.sql.SparkSession	getSparkSession()
protected Boolean	onInitializingWriteClient(HoodieWriteClient writeClient) Callback to initialize write client and start compaction service if required
protected Pair<CompletableFuture,ExecutorService>	startService() Service implementation

Methods inherited from class org.apache.hudi.utilities.deltastreamer.AbstractDeltaStreamerService

start

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DeltaSyncService

public DeltaSyncService(HoodieDeltaStreamer.Config cfg,
                        org.apache.spark.api.java.JavaSparkContext jssc,
                        org.apache.hadoop.fs.FileSystem fs,
                        org.apache.hadoop.hive.conf.HiveConf hiveConf)
                 throws IOException

Throws:

IOException

Method Detail

getDeltaSync

public DeltaSync getDeltaSync()

startService

protected Pair<CompletableFuture,ExecutorService> startService()

Description copied from class: AbstractDeltaStreamerService

Service implementation

Specified by:

startService in class AbstractDeltaStreamerService

Returns:

onInitializingWriteClient

protected Boolean onInitializingWriteClient(HoodieWriteClient writeClient)

Callback to initialize write client and start compaction service if required

Parameters:

writeClient - HoodieWriteClient

Returns:

close

public void close()

Close all resources

getSchemaProvider

public SchemaProvider getSchemaProvider()

getSparkSession

public org.apache.spark.sql.SparkSession getSparkSession()

getJavaSparkContext

public org.apache.spark.api.java.JavaSparkContext getJavaSparkContext()

getAsyncCompactService

public HoodieDeltaStreamer.AsyncCompactService getAsyncCompactService()

getProps

public TypedProperties getProps()