io.datanerds.verteiler

Class BlockingQueueConsumer<K,V>

java.lang.Object

io.datanerds.verteiler.BlockingQueueConsumer<K,V>

Type Parameters:

K - Key type

V - Value type

All Implemented Interfaces:

org.apache.kafka.clients.consumer.ConsumerRebalanceListener

public class BlockingQueueConsumer<K,V>
extends java.lang.Object
implements org.apache.kafka.clients.consumer.ConsumerRebalanceListener

This class leverages the kafka-clients consumer implementation to distribute messages from assigned partitions to BlockingQueues. Each assigned partition will relay its messages to its own queue. In addition, each queue has a consuming process/thread. Once a message has been "processed" successfully its offset will be marked to be committed.

Constructor Summary

Constructors

Constructor and Description
BlockingQueueConsumer(java.lang.String topic, java.util.Properties kafkaConfig, int queueSize, java.util.function.Consumer<V> action) This constructor immediately connects to the kafka broker and creates a Processor for each assigned partition.

Method Summary

Modifier and Type	Method and Description
void	onPartitionsAssigned(java.util.Collection<org.apache.kafka.common.TopicPartition> partitions)
void	onPartitionsRevoked(java.util.Collection<org.apache.kafka.common.TopicPartition> partitions)
void	start() Start consuming/ relaying messages to the processors.
void	stop() Stops all background activities: kafka message consumption, message relay and processing.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BlockingQueueConsumer

public BlockingQueueConsumer(java.lang.String topic,
                             java.util.Properties kafkaConfig,
                             int queueSize,
                             java.util.function.Consumer<V> action)

This constructor immediately connects to the kafka broker and creates a Processor for each assigned partition.

Parameters:

topic - The topic to subscribe to

kafkaConfig - Kafka client configuration

queueSize - Size of the BlockingQueues for each Processor

action - Consumer of the transported message

Throws:

java.lang.IllegalArgumentException - if problems are found with the config

Method Detail

start

public void start()

Start consuming/ relaying messages to the processors.

Throws:

java.lang.IllegalStateException - in case the consumer has been started before

stop

public void stop()

Stops all background activities: kafka message consumption, message relay and processing.

Throws:

java.lang.IllegalArgumentException - in case the consumer has not been started

onPartitionsAssigned

public void onPartitionsAssigned(java.util.Collection<org.apache.kafka.common.TopicPartition> partitions)

Specified by:

onPartitionsAssigned in interface org.apache.kafka.clients.consumer.ConsumerRebalanceListener

onPartitionsRevoked

public void onPartitionsRevoked(java.util.Collection<org.apache.kafka.common.TopicPartition> partitions)

Specified by:

onPartitionsRevoked in interface org.apache.kafka.clients.consumer.ConsumerRebalanceListener