zio.kafka.consumer.fetch
Members list
Type members
Classlikes
A fetch strategy determined which stream are allowed to fetch data in the next poll.
A fetch strategy determined which stream are allowed to fetch data in the next poll.
WARNING: this is an EXPERIMENTAL API and may change in an incompatible way without notice in any zio-kafka version.
Attributes
- Supertypes
-
class Objecttrait Matchableclass Any
- Known subtypes
A fetch strategy that allows a stream to fetch data when its queue size is at or below maxPartitionQueueSize, as long as the total queue size is at or below maxTotalQueueSize. This strategy is suitable when QueueSizeBasedFetchStrategy requires too much heap space, particularly when a lot of partitions are being consumed.
A fetch strategy that allows a stream to fetch data when its queue size is at or below maxPartitionQueueSize, as long as the total queue size is at or below maxTotalQueueSize. This strategy is suitable when QueueSizeBasedFetchStrategy requires too much heap space, particularly when a lot of partitions are being consumed.
Value parameters
- maxPartitionQueueSize
-
Maximum number of records to be buffered per partition. This buffer improves throughput and supports varying downstream message processing time, while maintaining some backpressure. Low values effectively disable prefetching in favour of low memory consumption. Large values leave it up to
maxTotalQueueSizeparameter to backpressure only over the buffers of all partitions together. The number of records that are fetched on every poll is controlled by themax.poll.recordssetting, the number of records fetched for every partition is somewhere between 0 andmax.poll.records. The default value for this parameter is 2 * the defaultmax.poll.recordsof 500, rounded to the nearest power of 2. - maxTotalQueueSize
-
Maximum number of records to be buffered over all partitions together. This can be used to limit memory usage when consuming a large number of partitions. When multiple streams are eligible for pre-fetching (because their queue size is below
maxPartitionQueueSize), but together they exceedmaxTotalQueueSize, then every call a random set of eligible streams is selected that stays belowmaxTotalQueueSize. The randomization ensures fairness and prevents read-starvation for streams at the end of the list. The default value is 20 * the default formaxPartitionQueueSize, allowing approximately 20 partitions to do pre-fetching in each poll.
Attributes
- Supertypes
-
trait Serializabletrait Producttrait Equalstrait FetchStrategyclass Objecttrait Matchableclass AnyShow all
A fetch strategy that allows a stream to fetch data when its queue size is at or below partitionPreFetchBufferLimit.
A fetch strategy that allows a stream to fetch data when its queue size is at or below partitionPreFetchBufferLimit.
Value parameters
- partitionPreFetchBufferLimit
-
The queue size at or below which more records are fetched and buffered (per partition). This buffer improves throughput and supports varying downstream message processing time, while maintaining some backpressure. Large values effectively disable backpressure at the cost of high memory usage, low values will effectively disable prefetching in favor of low memory consumption. The number of records that are fetched on every poll is controlled by the
max.poll.recordssetting, the number of records fetched for every partition is somewhere between 0 andmax.poll.records. The default value for this parameter is 2 * the defaultmax.poll.recordsof 500, rounded to the nearest power of 2. The value0disables pre-fetching.
Attributes
- Supertypes
-
trait Serializabletrait Producttrait Equalstrait FetchStrategyclass Objecttrait Matchableclass AnyShow all