org.apache.hudi.utilities.deltastreamer

Class SourceFormatAdapter

java.lang.Object

org.apache.hudi.utilities.deltastreamer.SourceFormatAdapter

public final class SourceFormatAdapter
extends Object

Adapts data-format provided by the source to the data-format required by the client (DeltaStreamer)

Constructor Summary

Constructors

Constructor and Description
SourceFormatAdapter(Source source)

Method Summary

Modifier and Type	Method and Description
InputBatch<org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>>	fetchNewDataInAvroFormat(Option<String> lastCkptStr, long sourceLimit) Fetch new data in avro format.
InputBatch<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>>	fetchNewDataInRowFormat(Option<String> lastCkptStr, long sourceLimit) Fetch new data in row format.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SourceFormatAdapter

public SourceFormatAdapter(Source source)

Method Detail

fetchNewDataInAvroFormat

public InputBatch<org.apache.spark.api.java.JavaRDD<org.apache.avro.generic.GenericRecord>> fetchNewDataInAvroFormat(Option<String> lastCkptStr,
                                                                                                                     long sourceLimit)

Fetch new data in avro format. If the source provides data in different format, they are translated to Avro format

Parameters:

lastCkptStr -

sourceLimit -

Returns:

fetchNewDataInRowFormat

public InputBatch<org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>> fetchNewDataInRowFormat(Option<String> lastCkptStr,
                                                                                                  long sourceLimit)

Fetch new data in row format. If the source provides data in different format, they are translated to Row format

Parameters:

lastCkptStr -

sourceLimit -

Returns: