org.apache.hudi.utilities.sources.helpers

Class IncrSourceHelper

java.lang.Object

org.apache.hudi.utilities.sources.helpers.IncrSourceHelper

public class IncrSourceHelper
extends Object

Constructor Summary

Constructors

Constructor and Description
IncrSourceHelper()

Method Summary

Modifier and Type	Method and Description
static Pair<String,String>	calculateBeginAndEndInstants(org.apache.spark.api.java.JavaSparkContext jssc, String srcBasePath, int numInstantsPerFetch, Option<String> beginInstant, boolean readLatestOnMissingBeginInstant) Find begin and end instants to be set for the next fetch
static void	validateInstantTime(org.apache.spark.sql.Row row, String instantTime, String sinceInstant, String endInstant) Validate instant time seen in the incoming row

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

IncrSourceHelper

public IncrSourceHelper()

Method Detail

calculateBeginAndEndInstants

public static Pair<String,String> calculateBeginAndEndInstants(org.apache.spark.api.java.JavaSparkContext jssc,
                                                               String srcBasePath,
                                                               int numInstantsPerFetch,
                                                               Option<String> beginInstant,
                                                               boolean readLatestOnMissingBeginInstant)

Find begin and end instants to be set for the next fetch

Parameters:

jssc - Java Spark Context

srcBasePath - Base path of Hudi source table

numInstantsPerFetch - Max Instants per fetch

beginInstant - Last Checkpoint String

readLatestOnMissingBeginInstant - when begin instant is missing, allow reading from latest committed instant

Returns:

begin and end instants

validateInstantTime

public static void validateInstantTime(org.apache.spark.sql.Row row,
                                       String instantTime,
                                       String sinceInstant,
                                       String endInstant)

Validate instant time seen in the incoming row

Parameters:

row - Input Row

instantTime - Hoodie Instant time of the row

sinceInstant - begin instant of the batch

endInstant - end instant of the batch