org.apache.hudi.utilities.deltastreamer

Class HoodieDeltaStreamer

java.lang.Object

org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer

All Implemented Interfaces:

Serializable

public class HoodieDeltaStreamer
extends Object
implements Serializable

An Utility which can incrementally take the output from HiveIncrementalPuller and apply it to the target dataset. Does not maintain any state, queries at runtime to see how far behind the target dataset is from the source dataset. This can be overriden to force sync from a timestamp. In continuous mode, DeltaStreamer runs in loop-mode going through the below operations (a) pull-from-source (b) write-to-sink (c) Schedule Compactions if needed (d) Conditionally Sync to Hive each cycle. For MOR table with continuous mode enabled, a seperate compactor thread is allocated to execute compactions

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HoodieDeltaStreamer.AsyncCompactService Async Compactor Service tha runs in separate thread.
static class	HoodieDeltaStreamer.Config
static class	HoodieDeltaStreamer.DeltaSyncService Syncs data either in single-run or in continuous mode.
static class	HoodieDeltaStreamer.Operation

Field Summary

Fields

Modifier and Type	Field and Description
static String	CHECKPOINT_KEY

Constructor Summary

Constructors

Constructor and Description
HoodieDeltaStreamer(HoodieDeltaStreamer.Config cfg, org.apache.spark.api.java.JavaSparkContext jssc)
HoodieDeltaStreamer(HoodieDeltaStreamer.Config cfg, org.apache.spark.api.java.JavaSparkContext jssc, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.hive.conf.HiveConf hiveConf)

Method Summary

Modifier and Type	Method and Description
HoodieDeltaStreamer.DeltaSyncService	getDeltaSyncService()
static void	main(String[] args)
void	shutdownGracefully()
void	sync() Main method to start syncing

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CHECKPOINT_KEY

public static String CHECKPOINT_KEY

Constructor Detail

HoodieDeltaStreamer

public HoodieDeltaStreamer(HoodieDeltaStreamer.Config cfg,
                           org.apache.spark.api.java.JavaSparkContext jssc)
                    throws IOException

Throws:

IOException

HoodieDeltaStreamer

public HoodieDeltaStreamer(HoodieDeltaStreamer.Config cfg,
                           org.apache.spark.api.java.JavaSparkContext jssc,
                           org.apache.hadoop.fs.FileSystem fs,
                           org.apache.hadoop.hive.conf.HiveConf hiveConf)
                    throws IOException

Throws:

IOException

Method Detail

shutdownGracefully

public void shutdownGracefully()

sync

public void sync()
          throws Exception

Main method to start syncing

Throws:

Exception

main

public static void main(String[] args)
                 throws Exception

Throws:

Exception

getDeltaSyncService

public HoodieDeltaStreamer.DeltaSyncService getDeltaSyncService()