org.apache.hudi.utilities.sources

Class Source<T>

java.lang.Object

org.apache.hudi.utilities.sources.Source<T>

All Implemented Interfaces:

Serializable

Direct Known Subclasses:

AvroSource, JsonSource, RowSource

public abstract class Source<T>
extends Object
implements Serializable

Represents a source from which we can tail data. Assumes a constructor that takes properties.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Source.SourceType

Field Summary

Fields

Modifier and Type	Field and Description
protected static org.apache.log4j.Logger	log
protected TypedProperties	props
protected org.apache.spark.api.java.JavaSparkContext	sparkContext
protected org.apache.spark.sql.SparkSession	sparkSession

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	Source(TypedProperties props, org.apache.spark.api.java.JavaSparkContext sparkContext, org.apache.spark.sql.SparkSession sparkSession, SchemaProvider schemaProvider)
protected	Source(TypedProperties props, org.apache.spark.api.java.JavaSparkContext sparkContext, org.apache.spark.sql.SparkSession sparkSession, SchemaProvider schemaProvider, Source.SourceType sourceType)

Method Summary

Modifier and Type	Method and Description
protected abstract InputBatch<T>	fetchNewData(Option<String> lastCkptStr, long sourceLimit)
InputBatch<T>	fetchNext(Option<String> lastCkptStr, long sourceLimit) Main API called by Hoodie Delta Streamer to fetch records
Source.SourceType	getSourceType()
org.apache.spark.sql.SparkSession	getSparkSession()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

log

protected static volatile org.apache.log4j.Logger log

props

protected transient TypedProperties props

sparkContext

protected transient org.apache.spark.api.java.JavaSparkContext sparkContext

sparkSession

protected transient org.apache.spark.sql.SparkSession sparkSession

Constructor Detail

Source

protected Source(TypedProperties props,
                 org.apache.spark.api.java.JavaSparkContext sparkContext,
                 org.apache.spark.sql.SparkSession sparkSession,
                 SchemaProvider schemaProvider)

Source

protected Source(TypedProperties props,
                 org.apache.spark.api.java.JavaSparkContext sparkContext,
                 org.apache.spark.sql.SparkSession sparkSession,
                 SchemaProvider schemaProvider,
                 Source.SourceType sourceType)

Method Detail

fetchNewData

protected abstract InputBatch<T> fetchNewData(Option<String> lastCkptStr,
                                              long sourceLimit)

fetchNext

public final InputBatch<T> fetchNext(Option<String> lastCkptStr,
                                     long sourceLimit)

Main API called by Hoodie Delta Streamer to fetch records

Parameters:

lastCkptStr - Last Checkpoint

sourceLimit - Source Limit

Returns:

getSourceType

public Source.SourceType getSourceType()

getSparkSession

public org.apache.spark.sql.SparkSession getSparkSession()