Package org.apache.beam.sdk.io.kafka

Class CustomTimestampPolicyWithLimitedDelay<K,V>

java.lang.Object

org.apache.beam.sdk.io.kafka.TimestampPolicy<K,V>

org.apache.beam.sdk.io.kafka.CustomTimestampPolicyWithLimitedDelay<K,V>

public class CustomTimestampPolicyWithLimitedDelay<K,V>
extends TimestampPolicy<K,V>

A policy for custom record timestamps where timestamps within a partition are expected to be roughly monotonically increasing with a cap on out of order event delays (say 1 minute). The watermark at any time is '(Min(now(), Max(event timestamp so far)) - max delay)'. However, watermark is never set in future and capped to 'now - max delay'. In addition, watermark advanced to 'now - max delay' when a partition is idle.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.io.kafka.TimestampPolicy

TimestampPolicy.PartitionContext

Constructor Summary

Constructors

Constructor	Description
CustomTimestampPolicyWithLimitedDelay(org.apache.beam.sdk.transforms.SerializableFunction<KafkaRecord<K,V>,org.joda.time.Instant> timestampFunction, org.joda.time.Duration maxDelay, java.util.Optional<org.joda.time.Instant> previousWatermark)	A policy for custom record timestamps where timestamps are expected to be roughly monotonically increasing with out of order event delays less than maxDelay.

Method Summary

Modifier and Type	Method	Description
org.joda.time.Instant	getTimestampForRecord(TimestampPolicy.PartitionContext ctx, KafkaRecord<K,V> record)	Returns record timestamp (aka event time).
org.joda.time.Instant	getWatermark(TimestampPolicy.PartitionContext ctx)	Returns watermark for the partition.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CustomTimestampPolicyWithLimitedDelay

public CustomTimestampPolicyWithLimitedDelay(org.apache.beam.sdk.transforms.SerializableFunction<KafkaRecord<K,V>,org.joda.time.Instant> timestampFunction,
                                             org.joda.time.Duration maxDelay,
                                             java.util.Optional<org.joda.time.Instant> previousWatermark)

A policy for custom record timestamps where timestamps are expected to be roughly monotonically increasing with out of order event delays less than maxDelay. The watermark at any time is Min(now(), max_event_timestamp) - maxDelay.

Parameters:

timestampFunction - A function to extract timestamp from the record

maxDelay - For any record in the Kafka partition, the timestamp of any subsequent record is expected to be after current record timestamp - maxDelay.

previousWatermark - Latest check-pointed watermark, see TimestampPolicyFactory.createTimestampPolicy(TopicPartition, Optional)

Method Detail

getTimestampForRecord

public org.joda.time.Instant getTimestampForRecord(TimestampPolicy.PartitionContext ctx,
                                                   KafkaRecord<K,V> record)

Description copied from class: TimestampPolicy

Returns record timestamp (aka event time). This is often based on the timestamp of the Kafka record. This is invoked for each record when it is processed in the reader.

Specified by:

getTimestampForRecord in class TimestampPolicy<K,V>

getWatermark

public org.joda.time.Instant getWatermark(TimestampPolicy.PartitionContext ctx)

Description copied from class: TimestampPolicy

Returns watermark for the partition. It is the timestamp before or at the timestamps of all future records consumed from the partition. See UnboundedSource.UnboundedReader.getWatermark() for more guidance on watermarks. E.g. if the record timestamp is 'LogAppendTime', watermark would be the timestamp of the last record since 'LogAppendTime' monotonically increases within a partition.

Specified by:

getWatermark in class TimestampPolicy<K,V>