org.apache.hudi.utilities.sources.helpers

Class KafkaOffsetGen.CheckpointUtils

java.lang.Object

org.apache.hudi.utilities.sources.helpers.KafkaOffsetGen.CheckpointUtils

Enclosing class:

KafkaOffsetGen

public static class KafkaOffsetGen.CheckpointUtils
extends Object

Constructor Summary

Constructors

Constructor and Description
CheckpointUtils()

Method Summary

Modifier and Type	Method and Description
static org.apache.spark.streaming.kafka.OffsetRange[]	computeOffsetRanges(HashMap<kafka.common.TopicAndPartition,org.apache.spark.streaming.kafka.KafkaCluster.LeaderOffset> fromOffsetMap, HashMap<kafka.common.TopicAndPartition,org.apache.spark.streaming.kafka.KafkaCluster.LeaderOffset> toOffsetMap, long numEvents) Compute the offset ranges to read from Kafka, while handling newly added partitions, skews, event limits.
static String	offsetsToStr(org.apache.spark.streaming.kafka.OffsetRange[] ranges) String representation of checkpoint
static HashMap<kafka.common.TopicAndPartition,org.apache.spark.streaming.kafka.KafkaCluster.LeaderOffset>	strToOffsets(String checkpointStr) Reconstruct checkpoint from string.
static long	totalNewMessages(org.apache.spark.streaming.kafka.OffsetRange[] ranges)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CheckpointUtils

public CheckpointUtils()

Method Detail

strToOffsets

public static HashMap<kafka.common.TopicAndPartition,org.apache.spark.streaming.kafka.KafkaCluster.LeaderOffset> strToOffsets(String checkpointStr)

Reconstruct checkpoint from string.

offsetsToStr

public static String offsetsToStr(org.apache.spark.streaming.kafka.OffsetRange[] ranges)

String representation of checkpoint

Format: topic1,0:offset0,1:offset1,2:offset2, .....

computeOffsetRanges

public static org.apache.spark.streaming.kafka.OffsetRange[] computeOffsetRanges(HashMap<kafka.common.TopicAndPartition,org.apache.spark.streaming.kafka.KafkaCluster.LeaderOffset> fromOffsetMap,
                                                                                 HashMap<kafka.common.TopicAndPartition,org.apache.spark.streaming.kafka.KafkaCluster.LeaderOffset> toOffsetMap,
                                                                                 long numEvents)

Compute the offset ranges to read from Kafka, while handling newly added partitions, skews, event limits.

Parameters:

fromOffsetMap - offsets where we left off last time

toOffsetMap - offsets of where each partitions is currently at

numEvents - maximum number of events to read.

totalNewMessages

public static long totalNewMessages(org.apache.spark.streaming.kafka.OffsetRange[] ranges)