Package io.smallrye.reactive.messaging.kafka.companion

Class ConsumerBuilder<K,​V>

  • Type Parameters:
    K - The record key type
    V - The record value type
    All Implemented Interfaces:
    Closeable, AutoCloseable, org.apache.kafka.clients.consumer.ConsumerRebalanceListener
    public class ConsumerBuilder<K,​V>
extends Object
implements org.apache.kafka.clients.consumer.ConsumerRebalanceListener, Closeable
    Kafka consumer wrapper for creating tasks (ConsumerTask) that consume records.

    The wrapped Kafka consumer is created lazily when a ConsumerTask is created and started. Until the consumer is created builder methods prefixed with with- act to provide additional configuration.

    The created consumer is closed automatically when the task is terminated. The consumer uses an underlying ExecutorService for polling records. As Kafka consumer is not thread-safe, other operations accessing the underlying consumer are also run inside the ExecutorService

    • Constructor Detail

      • ConsumerBuilder

        public ConsumerBuilder​(Map<String,​Object> props,
                       Duration kafkaApiTimeout,
                       org.apache.kafka.common.serialization.Deserializer<K> keyDeser,
                       org.apache.kafka.common.serialization.Deserializer<V> valueDeser)
        Creates a new ConsumerBuilder.

        Note that on producer creation Deserializer.configure(java.util.Map<java.lang.String, ?>, boolean) methods will NOT be called.

        Parameters:
        props - the initial properties for consumer creation
        kafkaApiTimeout - the timeout for api calls to Kafka
        keyDeser - the deserializer for record keys
        valueDeser - the deserializer for record values

      • ConsumerBuilder

        public ConsumerBuilder​(Map<String,​Object> props,
                       Duration kafkaApiTimeout,
                       String keyDeserializerType,
                       String valueDeserializerType)
        Creates a new ConsumerBuilder.

        On consumer creation, key and value deserializers will be created and Deserializer.configure(java.util.Map<java.lang.String, ?>, boolean) methods will be called.

        Parameters:
        props - the initial properties for consumer creation
        kafkaApiTimeout - the timeout for api calls to Kafka
        keyDeserializerType - the deserializer class name for record keys
        valueDeserializerType - the deserializer class name for record values

    • Method Detail

      • unwrap

        public org.apache.kafka.clients.consumer.KafkaConsumer<K,​V> unwrap()
        Returns:
        the underlying KafkaConsumer, may be null if producer is not yet created.

      • onPartitionsRevoked

        public void onPartitionsRevoked​(Collection<org.apache.kafka.common.TopicPartition> partitions)
        Specified by:
        onPartitionsRevoked in interface org.apache.kafka.clients.consumer.ConsumerRebalanceListener

      • onPartitionsAssigned

        public void onPartitionsAssigned​(Collection<org.apache.kafka.common.TopicPartition> partitions)
        Specified by:
        onPartitionsAssigned in interface org.apache.kafka.clients.consumer.ConsumerRebalanceListener

      • terminate

        public void terminate​(org.apache.kafka.clients.consumer.KafkaConsumer<K,​V> consumer,
                      Throwable throwable)

      • clientId

        public String clientId()
        Returns:
        the client.id property of this consumer

      • groupId

        public String groupId()
        Returns:
        the group.id property of this consumer

      • withProp

        public ConsumerBuilder<K,​V> withProp​(String key,
                                           String value)
        Add property to the configuration to be used when the consumer is created.
        Parameters:
        key - the key for property
        value - the value for property
        Returns:
        this ConsumerBuilder

      • withProps

        public ConsumerBuilder<K,​V> withProps​(Map<String,​Object> properties)
        Add properties to the configuration to be used when the consumer is created.
        Parameters:
        properties - the properties
        Returns:
        this ConsumerBuilder

      • withOffsetReset

        public ConsumerBuilder<K,​V> withOffsetReset​(org.apache.kafka.clients.consumer.OffsetResetStrategy offsetResetStrategy)
        Add configuration property for auto.offset.reset
        Parameters:
        offsetResetStrategy - the offset reset strategy
        Returns:
        this ConsumerBuilder

      • withIsolationLevel

        public ConsumerBuilder<K,​V> withIsolationLevel​(org.apache.kafka.common.IsolationLevel isolationLevel)
        Set isolation level
        Parameters:
        isolationLevel - the isolation level
        Returns:
        this ConsumerBuilder

      • withCommitAsyncWhen

        public ConsumerBuilder<K,​V> withCommitAsyncWhen​(Predicate<org.apache.kafka.clients.consumer.ConsumerRecord<K,​V>> commitPredicate)
        If set the given function is used to determine when to commit async
        Parameters:
        commitPredicate - the commit async predicate
        Returns:
        this ConsumerBuilder

      • withCommitAsync

        public ConsumerBuilder<K,​V> withCommitAsync​(Function<org.apache.kafka.clients.consumer.ConsumerRecord<K,​V>,​Map<org.apache.kafka.common.TopicPartition,​org.apache.kafka.clients.consumer.OffsetAndMetadata>> commitFunction,
                                                  org.apache.kafka.clients.consumer.OffsetCommitCallback callback)
        If set the given function is used to determine when and which offsets to commit async
        Parameters:
        commitFunction - the function to call for commit async
        callback - the callback to set for commit async
        Returns:
        this ConsumerBuilder

      • withCommitSyncWhen

        public ConsumerBuilder<K,​V> withCommitSyncWhen​(Predicate<org.apache.kafka.clients.consumer.ConsumerRecord<K,​V>> commitPredicate)
        If set the given function is used to determine when to commit sync
        Parameters:
        commitPredicate - the commit sync predicate
        Returns:
        this ConsumerBuilder

      • withCommitSync

        public ConsumerBuilder<K,​V> withCommitSync​(Function<org.apache.kafka.clients.consumer.ConsumerRecord<K,​V>,​Map<org.apache.kafka.common.TopicPartition,​org.apache.kafka.clients.consumer.OffsetAndMetadata>> commitFunction)
        If set the given function is used to determine when and which offsets to commit sync
        Parameters:
        commitFunction - the function to call
        Returns:
        this ConsumerBuilder

      • withOnPartitionsAssigned

        public ConsumerBuilder<K,​V> withOnPartitionsAssigned​(Consumer<Collection<org.apache.kafka.common.TopicPartition>> onPartitionsAssigned)
        Parameters:
        onPartitionsAssigned - the callback to be called on partitions assigned
        Returns:
        this ConsumerBuilder

      • withOnPartitionsRevoked

        public ConsumerBuilder<K,​V> withOnPartitionsRevoked​(Consumer<Collection<org.apache.kafka.common.TopicPartition>> onPartitionsRevoked)
        Parameters:
        onPartitionsRevoked - the callback to be called on partitions revoked
        Returns:
        this ConsumerBuilder

      • withOnTermination

        public ConsumerBuilder<K,​V> withOnTermination​(BiConsumer<org.apache.kafka.clients.consumer.KafkaConsumer<K,​V>,​Throwable> onTermination)
        Parameters:
        onTermination - the callback to be called on termination
        Returns:
        this ConsumerBuilder

      • groupMetadata

        public org.apache.kafka.clients.consumer.ConsumerGroupMetadata groupMetadata()
        Returns:
        the consumer group metadata

      • currentAssignment

        public Set<org.apache.kafka.common.TopicPartition> currentAssignment()
        Returns:
        the current partition assignments of this consumer

      • assignment

        public Set<org.apache.kafka.common.TopicPartition> assignment()
        Returns:
        the partition assignment of this consumer

      • waitForAssignment

        public io.smallrye.mutiny.Uni<Set<org.apache.kafka.common.TopicPartition>> waitForAssignment()
        Returns:
        the Uni for waiting on the assignments of partitions

      • position

        public long position​(org.apache.kafka.common.TopicPartition partition)
        Parameters:
        partition - the topic partition
        Returns:
        the position of this consumer for the given topic partition

      • position

        public Map<org.apache.kafka.common.TopicPartition,​org.apache.kafka.clients.consumer.OffsetAndMetadata> position()
        Returns:
        the position of this consumer for all assigned topic partitions

      • resetToLastCommittedPositions

        public void resetToLastCommittedPositions()
        Reset to the last committed position

      • committed

        public org.apache.kafka.clients.consumer.OffsetAndMetadata committed​(org.apache.kafka.common.TopicPartition partition)
        Parameters:
        partition - the topic partition
        Returns:
        the map of committed offsets for the topic partition

      • committed

        public Map<org.apache.kafka.common.TopicPartition,​org.apache.kafka.clients.consumer.OffsetAndMetadata> committed​(org.apache.kafka.common.TopicPartition... partitions)
        Parameters:
        partitions - the topic partitions
        Returns:
        the map of committed offsets for topic partitions

      • committed

        public Map<org.apache.kafka.common.TopicPartition,​org.apache.kafka.clients.consumer.OffsetAndMetadata> committed()
        Returns:
        the map of committed offsets for topic partitions assigned to this consumer

      • pause

        public void pause()
        Pause the consumer

      • resume

        public void resume()
        Resume the consumer

      • fromOffsets

        public ConsumerTask<K,​V> fromOffsets​(Map<org.apache.kafka.common.TopicPartition,​Long> offsets,
                                           Function<io.smallrye.mutiny.Multi<org.apache.kafka.clients.consumer.ConsumerRecord<K,​V>>,​io.smallrye.mutiny.Multi<org.apache.kafka.clients.consumer.ConsumerRecord<K,​V>>> plugFunction)
        Create ConsumerTask for consuming records starting from the given offsets.

        The plug function is used to modify the Multi generating the records, ex. limiting the number of records produced.

        The task will run until a failure occurs or is explicitly stopped (subscription to the Multi cancelled).

        Parameters:
        offsets - the map of offsets to start consuming records
        plugFunction - the function to apply to the resulting multi
        Returns:
        the ConsumerTask

      • fromOffsets

        public ConsumerTask<K,​V> fromOffsets​(Map<org.apache.kafka.common.TopicPartition,​Long> offsets)
        Create ConsumerTask for consuming records starting from the given offsets.

        The task will run until a failure occurs or is explicitly stopped (subscription to the Multi cancelled).

        Parameters:
        offsets - the map of offsets to start consuming records
        Returns:
        the ConsumerTask

      • fromOffsets

        public ConsumerTask<K,​V> fromOffsets​(Map<org.apache.kafka.common.TopicPartition,​Long> offsets,
                                           Duration during)
        Create ConsumerTask for consuming records starting from the given offsets.

        The resulting ConsumerTask will be already started. The task will run until the Multi is terminated (with completion or failure) or is explicitly stopped (subscription to the Multi cancelled).

        Parameters:
        offsets - the map of offsets to start consuming records
        during - the duration of the producing task to run
        Returns:
        the ConsumerTask

      • fromOffsets

        public ConsumerTask<K,​V> fromOffsets​(Map<org.apache.kafka.common.TopicPartition,​Long> offsets,
                                           long numberOfRecords)
        Create ConsumerTask for consuming records starting from the given offsets.

        The resulting ConsumerTask will be already started. The task will run for generating the given number of records.

        Parameters:
        offsets - the map of offsets to start consuming records
        numberOfRecords - the number of records to produce
        Returns:
        the ConsumerTask

      • fromTopics

        public ConsumerTask<K,​V> fromTopics​(Set<String> topics,
                                          Function<io.smallrye.mutiny.Multi<org.apache.kafka.clients.consumer.ConsumerRecord<K,​V>>,​io.smallrye.mutiny.Multi<org.apache.kafka.clients.consumer.ConsumerRecord<K,​V>>> plugFunction)
        Create ConsumerTask for consuming records from the given topics.

        The plug function is used to modify the Multi generating the records, ex. limiting the number of records produced.

        The task will run until a failure occurs or is explicitly stopped (subscription to the Multi cancelled).

        Parameters:
        topics - the set of topics to subscribe
        plugFunction - the function to apply to the resulting multi
        Returns:
        the ConsumerTask

      • fromTopics

        public ConsumerTask<K,​V> fromTopics​(String topic,
                                          Function<io.smallrye.mutiny.Multi<org.apache.kafka.clients.consumer.ConsumerRecord<K,​V>>,​io.smallrye.mutiny.Multi<org.apache.kafka.clients.consumer.ConsumerRecord<K,​V>>> plugFunction)
        Create ConsumerTask for consuming records from the given topic.

        The plug function is used to modify the Multi generating the records, ex. limiting the number of records produced.

        The resulting ConsumerTask will be already started. The task will run until the Multi is terminated (with completion or failure) or is explicitly stopped (subscription to the Multi cancelled).

        Parameters:
        topic - the topic to subscribe
        plugFunction - the function to apply to the resulting multi
        Returns:
        the ConsumerTask

      • fromTopics

        public ConsumerTask<K,​V> fromTopics​(String... topics)
        Create ConsumerTask for consuming records from the given topics.

        The resulting ConsumerTask will be already started. The task will run until a failure occurs or is explicitly stopped (subscription to the Multi cancelled).

        Parameters:
        topics - the set of topics to subscribe
        Returns:
        the ConsumerTask

      • fromTopics

        public ConsumerTask<K,​V> fromTopics​(Set<String> topics)
        Create ConsumerTask for consuming records from the given topics.

        The resulting ConsumerTask will be already started. The task will run until a failure occurs or is explicitly stopped (subscription to the Multi cancelled).

        Parameters:
        topics - the set of topics to subscribe
        Returns:
        the ConsumerTask

      • fromTopics

        public ConsumerTask<K,​V> fromTopics​(Set<String> topics,
                                          long numberOfRecords)
        Create ConsumerTask for consuming records from the given topics.

        The resulting ConsumerTask will be already started. The task will run until the given number of records consumed.

        Parameters:
        topics - the set of topics to subscribe
        numberOfRecords - the number of records to produce
        Returns:
        the ConsumerTask

      • fromTopics

        public ConsumerTask<K,​V> fromTopics​(Set<String> topics,
                                          Duration during)
        Create ConsumerTask for consuming records from the given topics.

        The resulting ConsumerTask will be already started. The task will run during the given duration.

        Parameters:
        topics - the set of topics to subscribe
        during - the duration of the producing task to run
        Returns:
        the ConsumerTask

      • fromTopics

        public ConsumerTask<K,​V> fromTopics​(String topic,
                                          long numberOfRecords)
        Create ConsumerTask for consuming records from the given topic.

        The resulting ConsumerTask will be already started. The task will run until the given number of records consumed.

        Parameters:
        topic - the topic to subscribe
        numberOfRecords - the number of records to produce
        Returns:
        the ConsumerTask

      • fromTopics

        public ConsumerTask<K,​V> fromTopics​(String topic,
                                          Duration during)
        Create ConsumerTask for consuming records from the given topic.

        The resulting ConsumerTask will be already started. The task will run during the given duration.

        Parameters:
        topic - the topic to subscribe
        during - the duration of the producing task to run
        Returns:
        the ConsumerTask

      • fromTopics

        public ConsumerTask<K,​V> fromTopics​(String topic,
                                          long numberOfRecords,
                                          Duration during)
        Create ConsumerTask for consuming records from the given topic.

        The resulting ConsumerTask will be already started. The task will run until the given number of records consumed or during the given duration.

        Parameters:
        topic - the topic to subscribe
        numberOfRecords - the number of records to produce
        during - the duration of the producing task to run
        Returns:
        the ConsumerTask

      • fromPrevious

        public ConsumerTask<K,​V> fromPrevious​(KafkaTask<?,​?> task,
                                            Function<io.smallrye.mutiny.Multi<org.apache.kafka.clients.consumer.ConsumerRecord<K,​V>>,​io.smallrye.mutiny.Multi<org.apache.kafka.clients.consumer.ConsumerRecord<K,​V>>> plugFunction)

      • commitAndClose

        public void commitAndClose​(Map<org.apache.kafka.common.TopicPartition,​org.apache.kafka.clients.consumer.OffsetAndMetadata> offsets)
        Commit the given offset and close the consumer
        Parameters:
        offsets - the map of offsets to commit

      • commitAndClose

        public void commitAndClose()
        Commit the current offset and close the consumer