Package org.apache.beam.sdk.io.kafka

Interface TimestampPolicyFactory<KeyT,ValueT>

All Superinterfaces:

java.io.Serializable

Functional Interface:

This is a functional interface and can therefore be used as the assignment target for a lambda expression or method reference.

@FunctionalInterface
public interface TimestampPolicyFactory<KeyT,ValueT>
extends java.io.Serializable

An extendable factory to create a TimestampPolicy for each partition at runtime by KafkaIO reader. Subclasses implement createTimestampPolicy(org.apache.kafka.common.TopicPartition, java.util.Optional<org.joda.time.Instant>), which is invoked by the reader while starting or resuming from a checkpoint. Two commonly used policies are provided. See withLogAppendTime() and withProcessingTime().

Nested Class Summary

Nested Classes

Modifier and Type	Interface	Description
static class	TimestampPolicyFactory.LogAppendTimePolicy<K,V>	Assigns Kafka's log append time (server side ingestion time) to each record.
static class	TimestampPolicyFactory.ProcessingTimePolicy<K,V>	A simple policy that uses current time for event time and watermark.
static class	TimestampPolicyFactory.TimestampFnPolicy<K,V>	Internal policy to support deprecated withTimestampFn API.

Method Summary

Modifier and Type	Method	Description
TimestampPolicy<KeyT,ValueT>	createTimestampPolicy(org.apache.kafka.common.TopicPartition tp, java.util.Optional<org.joda.time.Instant> previousWatermark)	Creates a TimestampPolicy for a partition.
static <K,V> TimestampPolicyFactory<K,V>	withCreateTime(org.joda.time.Duration maxDelay)	CustomTimestampPolicyWithLimitedDelay using KafkaTimestampType.CREATE_TIME from the record for timestamp.
static <K,V> TimestampPolicyFactory<K,V>	withLogAppendTime()	A TimestampPolicy that assigns Kafka's log append time (server side ingestion time) to each record.
static <K,V> TimestampPolicyFactory<K,V>	withProcessingTime()	A TimestampPolicy that assigns processing time to each record.
static <K,V> TimestampPolicyFactory<K,V>	withTimestampFn(org.apache.beam.sdk.transforms.SerializableFunction<KafkaRecord<K,V>,org.joda.time.Instant> timestampFn)	Deprecated. Use @CustomTimestampPolicyWithLimitedDelay.

Method Detail

createTimestampPolicy

TimestampPolicy<KeyT,ValueT> createTimestampPolicy(org.apache.kafka.common.TopicPartition tp,
                                                         java.util.Optional<org.joda.time.Instant> previousWatermark)

Creates a TimestampPolicy for a partition. This is invoked by the reader at the start or while resuming from previous checkpoint.

Parameters:

tp - The returned policy applies to records from this TopicPartition.

previousWatermark - The latest check-pointed watermark. This is set when the reader is resuming from a checkpoint. This is a good value to return by implementations of TimestampPolicy.getWatermark(PartitionContext) until a better watermark can be established as more records are read.

withProcessingTime

static <K,V> TimestampPolicyFactory<K,V> withProcessingTime()

A TimestampPolicy that assigns processing time to each record. Specifically, this is the timestamp when the record becomes 'current' in the reader. The watermark aways advances to current time.

withLogAppendTime

static <K,V> TimestampPolicyFactory<K,V> withLogAppendTime()

A TimestampPolicy that assigns Kafka's log append time (server side ingestion time) to each record. The watermark for each Kafka partition is the timestamp of the last record read. If a partition is idle, the watermark advances roughly to 'current time - 2 seconds'. See KafkaIO.Read.withLogAppendTime() for longer description.

withCreateTime

static <K,V> TimestampPolicyFactory<K,V> withCreateTime(org.joda.time.Duration maxDelay)

CustomTimestampPolicyWithLimitedDelay using KafkaTimestampType.CREATE_TIME from the record for timestamp. See KafkaIO.Read.withCreateTime(Duration) for more complete documentation.

withTimestampFn

@Deprecated
static <K,V> TimestampPolicyFactory<K,V> withTimestampFn(org.apache.beam.sdk.transforms.SerializableFunction<KafkaRecord<K,V>,org.joda.time.Instant> timestampFn)

Deprecated.

Use @CustomTimestampPolicyWithLimitedDelay.

Used by the Read transform to support old timestamp functions API. This exists only to support other deprecated API KafkaIO.Read.withTimestampFn(SerializableFunction).
TODO(rangadi): Make this package private or remove it. It was never meant to be public.