org.apache.camel.processor.idempotent.kafka

Class KafkaIdempotentRepository

java.lang.Object

org.apache.camel.support.service.BaseService

org.apache.camel.support.service.ServiceSupport

org.apache.camel.processor.idempotent.kafka.KafkaIdempotentRepository

All Implemented Interfaces:

AutoCloseable, org.apache.camel.CamelContextAware, org.apache.camel.Service, org.apache.camel.ShutdownableService, org.apache.camel.spi.IdempotentRepository, org.apache.camel.StatefulService, org.apache.camel.SuspendableService

@ManagedResource(description="Kafka IdempotentRepository")
public class KafkaIdempotentRepository
extends org.apache.camel.support.service.ServiceSupport
implements org.apache.camel.spi.IdempotentRepository, org.apache.camel.CamelContextAware

A Kafka topic-based implementation of IdempotentRepository. Uses a local cache of previously seen Message IDs. Mutations that come in via the (add(String)), or remove(String) method will update the local cache and broadcast the change in state on a Kafka topic to other instances. The cache is back-filled from the topic by a Kafka consumer. The topic used must be unique per logical repository (i.e. two routes de-duplicate using different repositories, and different topics). This class makes no assumptions about the number of partitions (it is designed to consume from all at the same time), or replication factor of the topic. Each repository instance that uses the topic (e.g. typically on different machines running in parallel) controls its own consumer group, so in a cluster of 10 Camel processes using the same topic each will control its own offset. On startup, the instance subscribes to the topic and rewinds the offset to the beginning, rebuilding the cache to the latest state. The cache will not be considered warmed up until one poll of pollDurationMs in length returns 0 records. Startup will not be completed until either the cache has warmed up, or 30 seconds go by; if the latter happens the idempotent repository may be in an inconsistent state until its consumer catches up to the end of the topic. To use, this repository must be placed in the Camel registry, either manually or by registration as a bean in Spring/Blueprint, as it is CamelContext aware.

Field Summary

Fields inherited from class org.apache.camel.support.service.BaseService

BUILDED, FAILED, INITIALIZED, INITIALIZING, lock, NEW, SHUTDOWN, SHUTTINGDOWN, STARTED, STARTING, status, STOPPED, STOPPING, SUSPENDED, SUSPENDING

Constructor Summary

Constructors

Constructor and Description
KafkaIdempotentRepository() No-op constructor for XML/property-based object initialisation.
KafkaIdempotentRepository(String topic, Properties consumerConfig, Properties producerConfig)
KafkaIdempotentRepository(String topic, Properties consumerConfig, Properties producerConfig, int maxCacheSize, int pollDurationMs)
KafkaIdempotentRepository(String topic, String bootstrapServers)
KafkaIdempotentRepository(String topic, String bootstrapServers, int maxCacheSize, int pollDurationMs)

Method Summary

Modifier and Type	Method and Description
boolean	add(String key)
void	clear()
boolean	confirm(String key)
boolean	contains(String key)
protected void	doStart()
protected void	doStop()
String	getBootstrapServers()
org.apache.camel.CamelContext	getCamelContext()
Properties	getConsumerConfig()
long	getDuplicateCount()
int	getMaxCacheSize()
int	getPollDurationMs()
Properties	getProducerConfig()
String	getTopic()
boolean	isPollerRunning()
boolean	remove(String key)
void	setBootstrapServers(String bootstrapServers) Sets the
void	setCamelContext(org.apache.camel.CamelContext camelContext)
void	setConsumerConfig(Properties consumerConfig) Sets the properties that will be used by the Kafka consumer.
void	setMaxCacheSize(int maxCacheSize) Sets the maximum size of the local key cache.
void	setPollDurationMs(int pollDurationMs) Sets the poll duration of the Kafka consumer.
void	setProducerConfig(Properties producerConfig) Sets the properties that will be used by the Kafka producer.
void	setTopic(String topic) Sets the name of the Kafka topic used by this idempotent repository.

Methods inherited from class org.apache.camel.support.service.BaseService

build, doBuild, doFail, doInit, doLifecycleChange, doResume, doShutdown, doSuspend, fail, getStatus, init, isBuild, isInit, isNew, isRunAllowed, isShutdown, isStarted, isStarting, isStartingOrStarted, isStopped, isStopping, isStoppingOrStopped, isSuspended, isSuspending, isSuspendingOrSuspended, resume, shutdown, start, stop, suspend

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.camel.spi.IdempotentRepository

add, confirm, contains, remove

Methods inherited from interface org.apache.camel.Service

build, close, init, start, stop

Methods inherited from interface org.apache.camel.CamelContextAware

trySetCamelContext

Methods inherited from interface org.apache.camel.StatefulService

getStatus, isRunAllowed, isStarted, isStarting, isStopped, isStopping, isSuspending

Methods inherited from interface org.apache.camel.SuspendableService

isSuspended, resume, suspend

Methods inherited from interface org.apache.camel.ShutdownableService

shutdown

Constructor Detail

KafkaIdempotentRepository

public KafkaIdempotentRepository()

No-op constructor for XML/property-based object initialisation. From Java, prefer one of the other constructors.

KafkaIdempotentRepository

public KafkaIdempotentRepository(String topic,
                                 String bootstrapServers)

KafkaIdempotentRepository

public KafkaIdempotentRepository(String topic,
                                 String bootstrapServers,
                                 int maxCacheSize,
                                 int pollDurationMs)

KafkaIdempotentRepository

public KafkaIdempotentRepository(String topic,
                                 Properties consumerConfig,
                                 Properties producerConfig)

KafkaIdempotentRepository

public KafkaIdempotentRepository(String topic,
                                 Properties consumerConfig,
                                 Properties producerConfig,
                                 int maxCacheSize,
                                 int pollDurationMs)

Method Detail

getTopic

public String getTopic()

setTopic

public void setTopic(String topic)

Sets the name of the Kafka topic used by this idempotent repository. Each functionally-separate repository should use a different topic.

Parameters:

topic - The topic name.

getBootstrapServers

public String getBootstrapServers()

setBootstrapServers

public void setBootstrapServers(String bootstrapServers)

Sets the

 bootstrap.servers
 

property on the internal Kafka producer and consumer. Use this as shorthand if not setting consumerConfig and producerConfig. If used, this component will apply sensible default configurations for the producer and consumer.

Parameters:

bootstrapServers - The

                         bootstrap.servers
                         

value to use.

getProducerConfig

public Properties getProducerConfig()

setProducerConfig

public void setProducerConfig(Properties producerConfig)

Sets the properties that will be used by the Kafka producer. Overrides bootstrapServers, so must define the

 bootstrap.servers
 

property itself. Prefer using bootstrapServers for default configuration unless you specifically need non-standard configuration options such as SSL/SASL.

Parameters:

producerConfig - The producer configuration properties.

getConsumerConfig

public Properties getConsumerConfig()

setConsumerConfig

public void setConsumerConfig(Properties consumerConfig)

Sets the properties that will be used by the Kafka consumer. Overrides bootstrapServers, so must define the

 bootstrap.servers
 

property itself. Prefer using bootstrapServers for default configuration unless you specifically need non-standard configuration options such as SSL/SASL.

Parameters:

consumerConfig - The consumer configuration properties.

getMaxCacheSize

public int getMaxCacheSize()

setMaxCacheSize

public void setMaxCacheSize(int maxCacheSize)

Sets the maximum size of the local key cache.

Parameters:

maxCacheSize - The maximum key cache size.

getPollDurationMs

public int getPollDurationMs()

setPollDurationMs

public void setPollDurationMs(int pollDurationMs)

Sets the poll duration of the Kafka consumer. The local caches are updated immediately; this value will affect how far behind other peers in the cluster are, which are updating their caches from the topic, relative to the idempotent consumer instance issued the cache action message. The default value of this is DEFAULT_POLL_DURATION_MS. If setting this value explicitly, be aware that there is a tradeoff between the remote cache liveness and the volume of network traffic between this repository's consumer and the Kafka brokers. The cache warmup process also depends on there being one poll that fetches nothing - this indicates that the stream has been consumed up to the current point. If the poll duration is excessively long for the rate at which messages are sent on the topic, there exists a possibility that the cache cannot be warmed up and will operate in an inconsistent state relative to its peers until it catches up.

Parameters:

pollDurationMs - The poll duration in milliseconds.

setCamelContext

public void setCamelContext(org.apache.camel.CamelContext camelContext)

Specified by:

setCamelContext in interface org.apache.camel.CamelContextAware

getCamelContext

public org.apache.camel.CamelContext getCamelContext()

Specified by:

getCamelContext in interface org.apache.camel.CamelContextAware

doStart

protected void doStart()
                throws Exception

Overrides:

doStart in class org.apache.camel.support.service.BaseService

Throws:

Exception

doStop

protected void doStop()

Overrides:

doStop in class org.apache.camel.support.service.BaseService

add

public boolean add(String key)

Specified by:

add in interface org.apache.camel.spi.IdempotentRepository

contains

@ManagedOperation(description="Does the store contain the given key")
public boolean contains(String key)

Specified by:

contains in interface org.apache.camel.spi.IdempotentRepository

remove

@ManagedOperation(description="Remove the key from the store")
public boolean remove(String key)

Specified by:

remove in interface org.apache.camel.spi.IdempotentRepository

confirm

public boolean confirm(String key)

Specified by:

confirm in interface org.apache.camel.spi.IdempotentRepository

clear

public void clear()

Specified by:

clear in interface org.apache.camel.spi.IdempotentRepository

getDuplicateCount

@ManagedOperation(description="Number of times duplicate messages have been detected")
public long getDuplicateCount()

isPollerRunning

@ManagedOperation(description="Number of times duplicate messages have been detected")
public boolean isPollerRunning()