org.apache.hudi.utilities.sources

Class RowSource

java.lang.Object

org.apache.hudi.utilities.sources.Source<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

org.apache.hudi.utilities.sources.RowSource

All Implemented Interfaces:

Serializable

Direct Known Subclasses:

HoodieIncrSource

public abstract class RowSource
extends Source<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hudi.utilities.sources.Source

Source.SourceType

Field Summary

Fields inherited from class org.apache.hudi.utilities.sources.Source

log, props, sparkContext, sparkSession

Constructor Summary

Constructors

Constructor and Description
RowSource(TypedProperties props, org.apache.spark.api.java.JavaSparkContext sparkContext, org.apache.spark.sql.SparkSession sparkSession, SchemaProvider schemaProvider)

Method Summary

Modifier and Type	Method and Description
protected InputBatch<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>	fetchNewData(Option<String> lastCkptStr, long sourceLimit)
protected abstract Pair<Option<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>,String>	fetchNextBatch(Option<String> lastCkptStr, long sourceLimit)

Methods inherited from class org.apache.hudi.utilities.sources.Source

fetchNext, getSourceType, getSparkSession

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RowSource

public RowSource(TypedProperties props,
                 org.apache.spark.api.java.JavaSparkContext sparkContext,
                 org.apache.spark.sql.SparkSession sparkSession,
                 SchemaProvider schemaProvider)

Method Detail

fetchNextBatch

protected abstract Pair<Option<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>,String> fetchNextBatch(Option<String> lastCkptStr,
                                                                                                              long sourceLimit)

fetchNewData

protected final InputBatch<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>> fetchNewData(Option<String> lastCkptStr,
                                                                                                long sourceLimit)

Specified by:

fetchNewData in class Source<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>