org.apache.gobblin.source.extractor

Interface Watermark

All Known Subinterfaces:

CheckpointableWatermark, CheckpointableWatermark, ComparableWatermark, Watermark

public interface Watermark

A Watermark represents a checkpoint in data processing, and indicates that all data up to specific point has been pulled from some source. Implementations of this interface are responsible for defining data structures in order to track this state.

A Watermark will be serialized in WorkUnits and WorkUnitStates. The toJson() method will be used to serialize the Watermark into a JsonElement.

Method Summary

Modifier and Type	Method and Description
short	calculatePercentCompletion(Watermark lowWatermark, Watermark highWatermark) This method must return a value from [0, 100].
com.google.gson.JsonElement	toJson() Convert this Watermark into a JsonElement.

Method Detail

toJson

com.google.gson.JsonElement toJson()

Convert this Watermark into a JsonElement.

Returns:

a JsonElement representing this Watermark.

calculatePercentCompletion

short calculatePercentCompletion(Watermark lowWatermark,
                                 Watermark highWatermark)

This method must return a value from [0, 100]. The value should correspond to a percent completion. Given two Watermark values, where the lowWatermark is the starting point, and the highWatermark is the goal, what is the percent completion of this Watermark.

Parameters:

lowWatermark - is the starting Watermark for the percent completion calculation. So if this.equals(lowWatermark) is true, this method should return 0.

highWatermark - is the end value Watermark for the percent completion calculation. So if this.equals(highWatermark) is true, this method should return 100.

Returns:

a value from [0, 100] representing the percentage completion of this Watermark.