zio-kafka
API
Generated with
zio-kafka/zio.kafka/zio.kafka.consumer/Consumer

Consumer

zio.kafka.consumer.Consumer
See theConsumer companion trait
object Consumer

Attributes

Companion
trait
Graph
Supertypes
class Object
trait Matchable
class Any
Self type
Consumer.type

Members list

Type members

Classlikes

object AutoOffsetStrategy

See ConsumerSettings.withOffsetRetrieval.

See ConsumerSettings.withOffsetRetrieval.

Attributes

Companion
trait
Supertypes
trait Sum
trait Mirror
class Object
trait Matchable
class Any
Self type
sealed trait AutoOffsetStrategy

Attributes

Companion
object
Supertypes
class Object
trait Matchable
class Any
Known subtypes
object Earliest
object Latest
object None
Self type
case object CommitTimeout extends RuntimeException, NoStackTrace

Attributes

Supertypes
trait Singleton
trait Product
trait Mirror
trait Product
trait Equals
trait NoStackTrace
class RuntimeException
class Exception
class Throwable
trait Serializable
class Object
trait Matchable
class Any
Show all
Self type
object OffsetRetrieval

Attributes

Companion
trait
Supertypes
trait Sum
trait Mirror
class Object
trait Matchable
class Any
Self type
sealed trait OffsetRetrieval

See ConsumerSettings.withOffsetRetrieval.

See ConsumerSettings.withOffsetRetrieval.

Attributes

Companion
object
Supertypes
class Object
trait Matchable
class Any
Known subtypes
class Auto
class Manual

Value members

Concrete methods

def assignment: RIO[Consumer, Set[TopicPartition]]

Accessor method

Accessor method

Attributes

def beginningOffsets(partitions: Set[TopicPartition], timeout: Duration): RIO[Consumer, Map[TopicPartition, Long]]

Accessor method

Accessor method

Attributes

def committed(partitions: Set[TopicPartition], timeout: Duration): RIO[Consumer, Map[TopicPartition, Option[OffsetAndMetadata]]]

Accessor method

Accessor method

Attributes

def consumeWith[R : Tag, R1 : Tag, K, V](settings: ConsumerSettings, subscription: Subscription, keyDeserializer: Deserializer[R, K], valueDeserializer: Deserializer[R, V], commitRetryPolicy: Schedule[Any, Any, Any])(f: (ConsumerRecord[K, V]) => URIO[R1, Unit]): RIO[R & R1, Unit]

Execute an effect for each record and commit the offset after processing

Execute an effect for each record and commit the offset after processing

This method is the easiest way of processing messages on a Kafka topic.

Messages on a single partition are processed sequentially, while the processing of multiple partitions happens in parallel.

Offsets are committed after execution of the effect. They are batched when a commit action is in progress to avoid backpressuring the stream. When commits fail due to a org.apache.kafka.clients.consumer.RetriableCommitFailedException they are retried according to commitRetryPolicy

The effect should absorb any failures. Failures should be handled by retries or ignoring the error, which will result in the Kafka message being skipped.

Messages are processed with 'at least once' consistency: it is not guaranteed that every message that is processed by the effect has a corresponding offset commit before stream termination.

Usage example:

val settings: ConsumerSettings = ???
val subscription = Subscription.Topics(Set("my-kafka-topic"))

val consumerIO = Consumer.consumeWith(settings, subscription, Serdes.string, Serdes.string) { record =>
 // Process the received record here
 putStrLn(s"Received record: ${record.key()}: ${record.value()}")
}

Type parameters

K

Type of keys (an implicit Deserializer should be in scope)

R

Environment for the consuming effect

R1

Environment for the deserializers

V

Type of values (an implicit Deserializer should be in scope)

Value parameters

commitRetryPolicy

Retry commits that failed due to a RetriableCommitFailedException according to this schedule

f

Function that returns the effect to execute for each message. It is passed the org.apache.kafka.clients.consumer.ConsumerRecord.

keyDeserializer

Deserializer for the key of the messages

settings

Settings for creating a Consumer

subscription

Topic subscription parameters

valueDeserializer

Deserializer for the value of the messages

Attributes

Returns

Effect that completes with a unit value only when interrupted. May fail when the Consumer fails.

def endOffsets(partitions: Set[TopicPartition], timeout: Duration): RIO[Consumer, Map[TopicPartition, Long]]

Accessor method

Accessor method

Attributes

def fromJavaConsumerWithPermit(javaConsumer: Consumer[Array[Byte], Array[Byte]], settings: ConsumerSettings, access: Semaphore, diagnostics: Diagnostics): ZIO[Scope, Throwable, Consumer]

Create a zio-kafka Consumer from an org.apache.kafka KafkaConsumer.

Create a zio-kafka Consumer from an org.apache.kafka KafkaConsumer.

You are responsible for all of the following:

  • creating and closing the KafkaConsumer,
  • making sure auto.commit is disabled,
  • creating access as a fair semaphore with a single permit,
  • acquire a permit from access before using the consumer, and release if afterwards,
  • not using the following consumer methods: subscribe, unsubscribe, assign, poll, commit*, seek, pause, resume, and enforceRebalance,
  • keeping the consumer config given to the java consumer in sync with the properties in settings (for example by constructing settings with ConsumerSettings(bootstrapServers).withProperties(config)).

Any deviation of these rules is likely to cause hard to track errors.

Semaphore access is shared between you and the zio-kafka consumer. Use it as short as possible; while you hold a permit the zio-kafka consumer is blocked.

Value parameters

access

A Semaphore with 1 permit.

diagnostics

Optional diagnostics listener

javaConsumer

Consumer

settings

Settings

Attributes

def listTopics(timeout: Duration): RIO[Consumer, Map[String, List[PartitionInfo]]]

Accessor method

Accessor method

Attributes

def make(settings: ConsumerSettings, diagnostics: Diagnostics): ZIO[Scope, Throwable, Consumer]

A new consumer.

A new consumer.

Value parameters

diagnostics

an optional callback for key events in the consumer life-cycle. The callbacks will be executed in a separate fiber. Since the events are queued, failure to handle these events leads to out of memory errors

Attributes

def metrics: RIO[Consumer, Map[MetricName, Metric]]

Accessor method

Accessor method

Attributes

def offsetsForTimes(timestamps: Map[TopicPartition, Long], timeout: Duration): RIO[Consumer, Map[TopicPartition, OffsetAndTimestamp]]

Accessor method

Accessor method

Attributes

def partitionedAssignmentStream[R, K, V](subscription: Subscription, keyDeserializer: Deserializer[R, K], valueDeserializer: Deserializer[R, V]): ZStream[Consumer, Throwable, Chunk[(TopicPartition, ZStream[R, Throwable, CommittableRecord[K, V]])]]

Accessor method

Accessor method

Attributes

def partitionedStream[R, K, V](subscription: Subscription, keyDeserializer: Deserializer[R, K], valueDeserializer: Deserializer[R, V]): ZStream[Consumer, Throwable, (TopicPartition, ZStream[R, Throwable, CommittableRecord[K, V]])]

Accessor method

Accessor method

Attributes

def partitionsFor(topic: String, timeout: Duration): RIO[Consumer, List[PartitionInfo]]

Accessor method

Accessor method

Attributes

def plainStream[R, K, V](subscription: Subscription, keyDeserializer: Deserializer[R, K], valueDeserializer: Deserializer[R, V], bufferSize: Int): ZStream[R & Consumer, Throwable, CommittableRecord[K, V]]

Accessor method

Accessor method

Attributes

def position(partition: TopicPartition, timeout: Duration): RIO[Consumer, Long]

Accessor method

Accessor method

Attributes

def stopConsumption: RIO[Consumer, Unit]

Accessor method

Accessor method

Attributes

def subscription: RIO[Consumer, Set[String]]

Accessor method

Accessor method

Attributes

Deprecated methods

def fromJavaConsumer(javaConsumer: Consumer[Array[Byte], Array[Byte]], settings: ConsumerSettings, diagnostics: Diagnostics): ZIO[Scope, Throwable, Consumer]

Create a zio-kafka Consumer from an org.apache.kafka KafkaConsumer

Create a zio-kafka Consumer from an org.apache.kafka KafkaConsumer

You are responsible for creating and closing the KafkaConsumer. Make sure auto.commit is disabled.

Attributes

Deprecated
[Since version 2.9.0]

Concrete fields

val offsetBatches: ZSink[Any, Nothing, Offset, Nothing, OffsetBatch]
Generated with