org.apache.hudi.utilities.deltastreamer

Class DeltaSync

java.lang.Object

org.apache.hudi.utilities.deltastreamer.DeltaSync

All Implemented Interfaces:

Serializable

public class DeltaSync
extends Object
implements Serializable

Sync's one batch of data to hoodie dataset

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static String	CHECKPOINT_KEY
static String	CHECKPOINT_RESET_KEY
protected static org.apache.log4j.Logger	log

Constructor Summary

Constructors

Constructor and Description
DeltaSync(HoodieDeltaStreamer.Config cfg, org.apache.spark.sql.SparkSession sparkSession, SchemaProvider schemaProvider, HoodieTableType tableType, TypedProperties props, org.apache.spark.api.java.JavaSparkContext jssc, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.hive.conf.HiveConf hiveConf, java.util.function.Function<HoodieWriteClient,Boolean> onInitializingHoodieWriteClient)

Method Summary

Modifier and Type	Method and Description
void	close() Close all resources
void	setupWriteClient() Note that depending on configs and source-type, schemaProvider could either be eagerly or lazily created.
Option<String>	syncOnce() Run one round of delta sync and return new compaction instant if one got scheduled

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

log

protected static volatile org.apache.log4j.Logger log

CHECKPOINT_KEY

public static String CHECKPOINT_KEY

CHECKPOINT_RESET_KEY

public static String CHECKPOINT_RESET_KEY

Constructor Detail

DeltaSync

public DeltaSync(HoodieDeltaStreamer.Config cfg,
                 org.apache.spark.sql.SparkSession sparkSession,
                 SchemaProvider schemaProvider,
                 HoodieTableType tableType,
                 TypedProperties props,
                 org.apache.spark.api.java.JavaSparkContext jssc,
                 org.apache.hadoop.fs.FileSystem fs,
                 org.apache.hadoop.hive.conf.HiveConf hiveConf,
                 java.util.function.Function<HoodieWriteClient,Boolean> onInitializingHoodieWriteClient)
          throws IOException

Throws:

IOException

Method Detail

syncOnce

public Option<String> syncOnce()
                        throws Exception

Run one round of delta sync and return new compaction instant if one got scheduled

Throws:

Exception

setupWriteClient

public void setupWriteClient()

Note that depending on configs and source-type, schemaProvider could either be eagerly or lazily created. SchemaProvider creation is a precursor to HoodieWriteClient and AsyncCompactor creation. This method takes care of this constraint.

close

public void close()

Close all resources