Execute an effect for each record and commit the offset after processing

This method is the easiest way of processing messages on a Kafka topic.

Messages on a single partition are processed sequentially, while the processing of multiple partitions happens in parallel.

Offsets are committed after execution of the effect. They are batched when a commit action is in progress to avoid backpressuring the stream. When commits fail due to an org.apache.kafka.clients.consumer.RetriableCommitFailedException they are retried according to commitRetryPolicy

The effect should absorb any failures. Failures should be handled by retries or ignoring the error, which will result in the Kafka message being skipped.

Messages are processed with 'at least once' consistency: it is not guaranteed that every message that is processed by the effect has a corresponding offset commit before stream termination.

Usage example:

val settings: ConsumerSettings = ???
val subscription = Subscription.Topics(Set("my-kafka-topic"))

val consumerIO = Consumer.consumeWith(settings, subscription, Serdes.string, Serdes.string) { record =>
  // Process the received record here
  putStrLn(s"Received record: ${record.key()}: ${record.value()}")
}

Create a zio-kafka Consumer from an org.apache.kafka KafkaConsumer.

You are responsible for all of the following:

• creating and closing the KafkaConsumer,
• making sure auto.commit is disabled,
• creating access as a fair semaphore with a single permit,
• acquire a permit from access before using the consumer, and release it afterward,
• not using the following consumer methods: subscribe, unsubscribe, assign, poll, commit*, seek, pause, resume, and enforceRebalance,
• keeping the consumer config given to the java consumer in sync with the properties in settings (for example by constructing settings with ConsumerSettings(bootstrapServers).withProperties(config)).

Any deviation of these rules is likely to cause hard to track errors.

Semaphore access is shared between you and the zio-kafka consumer. Use it as short as possible; while you hold a permit the zio-kafka consumer is blocked.

Takes a StreamControl for some stream and runs the given ZIO workflow on that stream such that, when interrupted, stops fetching records and gracefully waits for the ZIO workflow to complete.

This is useful for running streams from within your application's Main class, such that streams are cleanly stopped when the application is shutdown (for example by your container runtime).

WARNING: this is an EXPERIMENTAL API and may disappear or change in an incompatible way without notice in any zio-kafka version.