Package org.apache.beam.sdk.io.kafka

Class KafkaIO.Write<K,​V>

  • All Implemented Interfaces:
    java.io.Serializable, org.apache.beam.sdk.transforms.display.HasDisplayData
    Enclosing class:
    KafkaIO
    public abstract static class KafkaIO.Write<K,​V>
extends org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<K,​V>>,​org.apache.beam.sdk.values.PDone>
    A PTransform to write to a Kafka topic with KVs . See KafkaIO for more information on usage and configuration.
    See Also:
    Serialized Form

    • Field Detail

      • AUTOVALUE_CLASS

        public static final java.lang.Class<org.apache.beam.sdk.io.kafka.AutoValue_KafkaIO_Write> AUTOVALUE_CLASS

    • Constructor Detail

      • Write

        public Write()

    • Method Detail

      • withKeySerializer

        public KafkaIO.Write<K,​V> withKeySerializer​(java.lang.Class<? extends org.apache.kafka.common.serialization.Serializer<K>> keySerializer)
        Wrapper method over KafkaIO.WriteRecords.withKeySerializer(Class), used to keep the compatibility with old API based on KV type of element.

      • withProducerFactoryFn

        public KafkaIO.Write<K,​V> withProducerFactoryFn​(org.apache.beam.sdk.transforms.SerializableFunction<java.util.Map<java.lang.String,​java.lang.Object>,​org.apache.kafka.clients.producer.Producer<K,​V>> producerFactoryFn)
        Wrapper method over KafkaIO.WriteRecords.withProducerFactoryFn(SerializableFunction), used to keep the compatibility with old API based on KV type of element.

      • withConsumerFactoryFn

        public KafkaIO.Write<K,​V> withConsumerFactoryFn​(org.apache.beam.sdk.transforms.SerializableFunction<java.util.Map<java.lang.String,​java.lang.Object>,​? extends org.apache.kafka.clients.consumer.Consumer<?,​?>> consumerFactoryFn)
        Wrapper method over KafkaIO.WriteRecords.withConsumerFactoryFn(SerializableFunction), used to keep the compatibility with old API based on KV type of element.

      • updateProducerProperties

        @Deprecated
public KafkaIO.Write<K,​V> updateProducerProperties​(java.util.Map<java.lang.String,​java.lang.Object> configUpdates)
        Deprecated.
        as of version 2.13. Use withProducerConfigUpdates(Map) instead.
        Adds the given producer properties, overriding old values of properties with the same key.

      • withProducerConfigUpdates

        public KafkaIO.Write<K,​V> withProducerConfigUpdates​(java.util.Map<java.lang.String,​java.lang.Object> configUpdates)
        Update configuration for the producer. Note that the default producer properties will not be completely overridden. This method only updates the value which has the same key.

        By default, the producer uses the configuration from KafkaIO.WriteRecords.DEFAULT_PRODUCER_PROPERTIES.

      • withBadRecordErrorHandler

        public KafkaIO.Write<K,​V> withBadRecordErrorHandler​(org.apache.beam.sdk.transforms.errorhandling.ErrorHandler<org.apache.beam.sdk.transforms.errorhandling.BadRecord,​?> badRecordErrorHandler)
        Configure a ErrorHandler.BadRecordErrorHandler for sending records to if they fail to serialize when being sent to Kafka.

      • withGCPApplicationDefaultCredentials

        public KafkaIO.Write<K,​V> withGCPApplicationDefaultCredentials()
        Creates and sets the Application Default Credentials for a Kafka producer. This allows the consumer to be authenticated with a Google Kafka Server using OAuth.

      • expand

        public org.apache.beam.sdk.values.PDone expand​(org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<K,​V>> input)
        Specified by:
        expand in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<K,​V>>,​org.apache.beam.sdk.values.PDone>

      • validate

        public void validate​(@Nullable org.apache.beam.sdk.options.PipelineOptions options)
        Overrides:
        validate in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<K,​V>>,​org.apache.beam.sdk.values.PDone>

      • populateDisplayData

        public void populateDisplayData​(org.apache.beam.sdk.transforms.display.DisplayData.Builder builder)
        Specified by:
        populateDisplayData in interface org.apache.beam.sdk.transforms.display.HasDisplayData
        Overrides:
        populateDisplayData in class org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<K,​V>>,​org.apache.beam.sdk.values.PDone>

      • values

        public org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PCollection<V>,​org.apache.beam.sdk.values.PDone> values()
        Writes just the values to Kafka. This is useful for writing collections of values rather thank KVs.