Package org.apache.beam.sdk.io.kafka

Transforms for reading and writing from Apache Kafka.

Interface Summary

Interface	Description
CheckStopReadingFn
DeserializerProvider<T>	Provides a configured Deserializer instance and its associated Coder.
KafkaIO.Read.FakeFlinkPipelineOptions
KafkaMetrics	Stores and exports metrics for a batch of Kafka Client RPCs.
KafkaPublishTimestampFunction<T>	An interface for providing custom timestamp for elements written to Kafka.
TimestampPolicyFactory<KeyT,ValueT>	An extendable factory to create a TimestampPolicy for each partition at runtime by KafkaIO reader.

Class Summary

Class	Description
CheckStopReadingFnWrapper
ConfluentSchemaRegistryDeserializerProvider<T>	A DeserializerProvider that uses Confluent Schema Registry to resolve a Deserializers and Coder given a subject.
CustomTimestampPolicyWithLimitedDelay<K,V>	A policy for custom record timestamps where timestamps within a partition are expected to be roughly monotonically increasing with a cap on out of order event delays (say 1 minute).
KafkaCheckpointMark	Checkpoint for a KafkaUnboundedReader.
KafkaCheckpointMark.PartitionMark	A tuple to hold topic, partition, and offset that comprise the checkpoint for a single partition.
KafkaCommitOffset<K,V>	A PTransform that commits offsets of KafkaRecord.
KafkaIO	An unbounded source and a sink for Kafka topics.
KafkaIO.Read<K,V>	A PTransform to read from Kafka topics.
KafkaIO.Read.External	Exposes KafkaIO.TypedWithoutMetadata as an external transform for cross-language usage.
KafkaIO.Read.External.Configuration	Parameters class to expose the Read transform to an external SDK.
KafkaIO.ReadSourceDescriptors<K,V>	A PTransform to read from KafkaSourceDescriptor.
KafkaIO.TypedWithoutMetadata<K,V>	A PTransform to read from Kafka topics.
KafkaIO.Write<K,V>	A PTransform to write to a Kafka topic with KVs .
KafkaIO.Write.External	Exposes KafkaIO.Write as an external transform for cross-language usage.
KafkaIO.Write.External.Configuration	Parameters class to expose the Write transform to an external SDK.
KafkaIO.WriteRecords<K,V>	A PTransform to write to a Kafka topic with ProducerRecord's.
KafkaIOUtils	Common utility functions and default configurations for KafkaIO.Read and KafkaIO.ReadSourceDescriptors.
KafkaMetrics.KafkaMetricsImpl	Metrics of a batch of RPCs.
KafkaMetrics.NoOpKafkaMetrics	No-op implementation of KafkaResults.
KafkaReadSchemaTransformConfiguration	Configuration for reading from a Kafka topic.
KafkaReadSchemaTransformConfiguration.Builder	Builder for the KafkaReadSchemaTransformConfiguration.
KafkaReadSchemaTransformProvider
KafkaReadSchemaTransformProvider.ErrorFn
KafkaRecord<K,V>	KafkaRecord contains key and value of the record as well as metadata for the record (topic name, partition id, and offset).
KafkaRecordCoder<K,V>	Coder for KafkaRecord.
KafkaSchemaTransformTranslation
KafkaSchemaTransformTranslation.ReadRegistrar
KafkaSchemaTransformTranslation.WriteRegistrar
KafkaSinkMetrics	Helper class to create per worker metrics for Kafka Sink stages.
KafkaSourceDescriptor	Represents a Kafka source description.
KafkaWriteSchemaTransformProvider
KafkaWriteSchemaTransformProvider.KafkaWriteSchemaTransformConfiguration
KafkaWriteSchemaTransformProvider.KafkaWriteSchemaTransformConfiguration.Builder
ProducerRecordCoder<K,V>	Coder for ProducerRecord.
TimestampPolicy<K,V>	A timestamp policy to assign event time for messages in a Kafka partition and watermark for it.
TimestampPolicy.PartitionContext	The context contains state maintained in the reader for the partition.
TimestampPolicyFactory.LogAppendTimePolicy<K,V>	Assigns Kafka's log append time (server side ingestion time) to each record.
TimestampPolicyFactory.ProcessingTimePolicy<K,V>	A simple policy that uses current time for event time and watermark.
TimestampPolicyFactory.TimestampFnPolicy<K,V>	Internal policy to support deprecated withTimestampFn API.
TopicPartitionCoder	The Coder for encoding and decoding TopicPartition in Beam.

Enum Summary

Enum	Description
KafkaTimestampType	This is a copy of Kafka's TimestampType.