Package com.hazelcast.jet.kafka

Class KafkaDataConnection

java.lang.Object

com.hazelcast.dataconnection.DataConnectionBase

com.hazelcast.jet.kafka.KafkaDataConnection

All Implemented Interfaces:

DataConnection

public class KafkaDataConnection
extends DataConnectionBase

DataConnection implementation for Kafka.

KafkaDataConnection is usable both for sources and sinks.

Sources obtain KafkaConsumer instances using newConsumer(). Because the KafkaConsumer is not thread-safe so the DataConnection is used only to keep the connection metadata. A new instance is returned each time this method is called.

Sinks obtain KafkaProducer instances using getProducer(String). The producer instance may be either shared or single-use depending on the DataConnectionConfig.isShared() setting. Shared producer instance is limited to use with either no processing guarantees or at-least-once processing guarantees. Use with exactly-once is not possible.

The properties must be configured for both consumers and producers.

Since:

5.3

Constructor Summary

Constructors

Constructor	Description
KafkaDataConnection(DataConnectionConfig config)	Create KafkaDataConnection based on given config

Method Summary

Modifier and Type	Method	Description
void	destroy()
<K, V> org.apache.kafka.clients.producer.KafkaProducer<K,V>	getProducer(String transactionalId)	Returns an instance of KafkaProducer based on the DataConnection configuration.
<K, V> org.apache.kafka.clients.producer.KafkaProducer<K,V>	getProducer(String transactionalId, Properties properties)	Returns an instance of KafkaProducer based on the DataConnection configuration and given properties.
Collection<DataConnectionResource>	listResources()
<K, V> org.apache.kafka.clients.consumer.Consumer<K,V>	newConsumer()	Creates new instance of KafkaConsumer based on the DataConnection configuration.
<K, V> org.apache.kafka.clients.consumer.Consumer<K,V>	newConsumer(Properties properties)	Creates new instance of KafkaConsumer based on the DataConnection configuration and given properties parameter.
Collection<String>	resourceTypes()

Methods inherited from class com.hazelcast.dataconnection.DataConnectionBase

getConfig, getName, release, retain

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.hazelcast.dataconnection.DataConnection

options

Constructor Details

KafkaDataConnection

public KafkaDataConnection(@Nonnull
 DataConnectionConfig config)

Create KafkaDataConnection based on given config

Method Details

listResources

@Nonnull
public Collection<DataConnectionResource> listResources()

resourceTypes

@Nonnull
public Collection<String> resourceTypes()

newConsumer

@Nonnull
public <K,
V>
org.apache.kafka.clients.consumer.Consumer<K,V> newConsumer()

Creates new instance of KafkaConsumer based on the DataConnection configuration. Always creates a new instance of the consumer because KafkaConsumer is not thread-safe. The caller is responsible for closing the consumer instance.

Returns:

consumer instance

newConsumer

@Nonnull
public <K,
V>
org.apache.kafka.clients.consumer.Consumer<K,V> newConsumer(Properties properties)

Creates new instance of KafkaConsumer based on the DataConnection configuration and given properties parameter. Always creates a new instance of the consumer because KafkaConsumer is not thread-safe. The caller is responsible for closing the consumer instance.

Parameters:

properties - mapping properties to merge with data connection options.

Returns:

consumer instance

getProducer

@Nonnull
public <K,
V>
org.apache.kafka.clients.producer.KafkaProducer<K,V> getProducer(@Nullable
 String transactionalId)

Returns an instance of KafkaProducer based on the DataConnection configuration.

The caller is responsible for closing the producer instance. For non-shared producers the producer will be closed immediately upon calling close. For shared producers the producer will be closed when all users close the returned instance and this DataConnection is released by calling DataConnectionBase.release()

Parameters:

transactionalId - transaction id to pass as 'transactional.id' property to a new KafkaProducer instance, must be null for shared producer

getProducer

@Nonnull
public <K,
V>
org.apache.kafka.clients.producer.KafkaProducer<K,V> getProducer(@Nullable
 String transactionalId,
 @Nonnull
 Properties properties)

Returns an instance of KafkaProducer based on the DataConnection configuration and given properties.

The caller is responsible for closing the producer instance. For non-shared producers the producer will be closed immediately upon calling close. For shared producers the producer will be closed when all users close the returned instance and this DataConnection is released by calling DataConnectionBase.release()

Parameters:

transactionalId - transaction id to pass as 'transactional.id' property to a new KafkaProducer instance, must be null for shared producer

properties - properties. E.g, SQL mappings provide separate options, and they should be merged with data connection's properties. These properties have higher priority than data connection properties. If KafkaDataConnection is shared, then HazelcastException would be thrown.

destroy

public void destroy()