org.apache.gobblin.writer

Interface WatermarkStorage

@Alpha
public interface WatermarkStorage

Storage for Watermarks. Used in streaming execution.

Method Summary

Modifier and Type	Method and Description
void	commitWatermarks(Iterable<CheckpointableWatermark> watermarks) Commit a batch of watermarks to storage.
Map<String,CheckpointableWatermark>	getCommittedWatermarks(Class<? extends CheckpointableWatermark> watermarkClass, Iterable<String> sourcePartitions) Retrieve previously committed watermarks.

Method Detail

commitWatermarks

void commitWatermarks(Iterable<CheckpointableWatermark> watermarks)
               throws IOException

Commit a batch of watermarks to storage.

Throws:

IOException

getCommittedWatermarks

Map<String,CheckpointableWatermark> getCommittedWatermarks(Class<? extends CheckpointableWatermark> watermarkClass,
                                                           Iterable<String> sourcePartitions)
                                                    throws IOException

Retrieve previously committed watermarks.

Parameters:

watermarkClass: - the specific class that corresponds to the watermark expected

sourcePartitions: - a list of source partitions for whom we're retrieving committed watermarks.

Returns:

a map of String -> CheckpointableWatermark. The key corresponds to the source field in the CheckpointableWatermark and belongs to the list of source partitions passed in.

Throws:

IOException