org.apache.hudi.utilities.sources

Class HiveIncrPullSource

java.lang.Object

org.apache.hudi.utilities.sources.Source<org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>>

org.apache.hudi.utilities.sources.AvroSource

org.apache.hudi.utilities.sources.HiveIncrPullSource

All Implemented Interfaces:

Serializable

public class HiveIncrPullSource
extends AvroSource

Source to read deltas produced by HiveIncrementalPuller, commit by commit and apply to the target table

The general idea here is to have commits sync across the data pipeline.

[Source Tables(s)] ====> HiveIncrementalScanner ==> incrPullRootPath ==> targetTable {c1,c2,c3,...} {c1,c2,c3,...} {c1,c2,c3,...}

This produces beautiful causality, that makes data issues in ETLs very easy to debug

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hudi.utilities.sources.Source

Source.SourceType

Field Summary

Fields inherited from class org.apache.hudi.utilities.sources.Source

props, sparkContext, sparkSession

Constructor Summary

Constructors

Constructor and Description
HiveIncrPullSource(TypedProperties props, org.apache.spark.api.java.JavaSparkContext sparkContext, org.apache.spark.sql.SparkSession sparkSession, SchemaProvider schemaProvider)

Method Summary

Modifier and Type	Method and Description
protected InputBatch<org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>>	fetchNewData(Option<String> lastCheckpointStr, long sourceLimit)

Methods inherited from class org.apache.hudi.utilities.sources.Source

fetchNext, getSourceType, getSparkSession

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HiveIncrPullSource

public HiveIncrPullSource(TypedProperties props,
                          org.apache.spark.api.java.JavaSparkContext sparkContext,
                          org.apache.spark.sql.SparkSession sparkSession,
                          SchemaProvider schemaProvider)

Method Detail

fetchNewData

protected InputBatch<org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>> fetchNewData(Option<String> lastCheckpointStr,
                                                                                                            long sourceLimit)

Specified by:

fetchNewData in class Source<org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>>