public class LagBasedPartitionAssignor extends Object implements org.apache.kafka.clients.consumer.ConsumerPartitionAssignor, org.apache.kafka.common.Configurable
auto.offset.reset property. If auto.offset.reset=latest, we assign a
lag of 0. If auto.offset.reset=earliest (or any other value) we assume assign lag equal to the total number
of message currently available in that partition.
We then create a map storing the current total lag of all partitions assigned to each member of the consumer group.
Partitions are assigned in decreasing order of lag, with each partition assigned to the consumer with least total
number of assigned partitions, breaking ties by assigning to the consumer with the least total assigned lag.
Distributing partitions as evenly across consumers (by count) ensures that the partition assignment is balanced when
all partitions have a current lag of 0 or if the distribution of lags is heavily skewed. It also gives the consumer
group the best possible chance of remaining balanced if the assignment is retained for a long period.
For example, suppose there are two consumers C0 and C1, both subscribed to a topic t0 having 3 partitions with the
following lags:
t0p0: 100,000
t0p1: 50,000
t0p2: 60,000
The assignment will be:
C0: [t0p0]
C1: [t0p1, t0p2]
The total lag or partitions assigned to each consumer will be:
C0: 100,000
C1: 110,000
Compare this to the assignments made by the RangeAssignor:
C0: [t0p0, t0p1]
C1: [t0p2]
The RangeAssignor results in a less balanced total lag for each consumer of:
C0: 160,000
C1: 50,000ConsumerPartitionAssignor,
AbstractPartitionAssignor,
RangeAssignororg.apache.kafka.clients.consumer.ConsumerPartitionAssignor.Assignment, org.apache.kafka.clients.consumer.ConsumerPartitionAssignor.GroupAssignment, org.apache.kafka.clients.consumer.ConsumerPartitionAssignor.GroupSubscription, org.apache.kafka.clients.consumer.ConsumerPartitionAssignor.RebalanceProtocol, org.apache.kafka.clients.consumer.ConsumerPartitionAssignor.Subscription| Constructor and Description |
|---|
LagBasedPartitionAssignor() |
| Modifier and Type | Method and Description |
|---|---|
org.apache.kafka.clients.consumer.ConsumerPartitionAssignor.GroupAssignment |
assign(org.apache.kafka.common.Cluster metadata,
org.apache.kafka.clients.consumer.ConsumerPartitionAssignor.GroupSubscription subscriptions) |
void |
configure(Map<String,?> configs)
This class is instantiated by reflection, at which point Kafka passes in the consumer config properties for
the current instance via this method.
|
String |
name() |
public void configure(Map<String,?> configs)
configure in interface org.apache.kafka.common.Configurableconfigs - Kafka consumer configuration properties for the current instancepublic String name()
name in interface org.apache.kafka.clients.consumer.ConsumerPartitionAssignorpublic org.apache.kafka.clients.consumer.ConsumerPartitionAssignor.GroupAssignment assign(org.apache.kafka.common.Cluster metadata,
org.apache.kafka.clients.consumer.ConsumerPartitionAssignor.GroupSubscription subscriptions)
assign in interface org.apache.kafka.clients.consumer.ConsumerPartitionAssignorCopyright © 2017–2020 Grant Neale. All rights reserved.