org.apache.hudi.utilities.sources

Class HoodieIncrSource

java.lang.Object

org.apache.hudi.utilities.sources.Source<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

org.apache.hudi.utilities.sources.RowSource

org.apache.hudi.utilities.sources.HoodieIncrSource

All Implemented Interfaces:

Serializable

public class HoodieIncrSource
extends RowSource

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	HoodieIncrSource.Config

Nested classes/interfaces inherited from class org.apache.hudi.utilities.sources.Source

Source.SourceType

Field Summary

Fields inherited from class org.apache.hudi.utilities.sources.Source

log, props, sparkContext, sparkSession

Constructor Summary

Constructors

Constructor and Description
HoodieIncrSource(TypedProperties props, org.apache.spark.api.java.JavaSparkContext sparkContext, org.apache.spark.sql.SparkSession sparkSession, SchemaProvider schemaProvider)

Method Summary

Modifier and Type	Method and Description
Pair<Option<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>,String>	fetchNextBatch(Option<String> lastCkptStr, long sourceLimit)

Methods inherited from class org.apache.hudi.utilities.sources.RowSource

fetchNewData

Methods inherited from class org.apache.hudi.utilities.sources.Source

fetchNext, getSourceType, getSparkSession

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HoodieIncrSource

public HoodieIncrSource(TypedProperties props,
                        org.apache.spark.api.java.JavaSparkContext sparkContext,
                        org.apache.spark.sql.SparkSession sparkSession,
                        SchemaProvider schemaProvider)

Method Detail

fetchNextBatch

public Pair<Option<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>,String> fetchNextBatch(Option<String> lastCkptStr,
                                                                                                  long sourceLimit)

Specified by:

fetchNextBatch in class RowSource