com.github.grantneale.kafka

Class LagBasedPartitionAssignor

java.lang.Object

com.github.grantneale.kafka.LagBasedPartitionAssignor

All Implemented Interfaces:

org.apache.kafka.clients.consumer.ConsumerPartitionAssignor, org.apache.kafka.common.Configurable

public class LagBasedPartitionAssignor
extends Object
implements org.apache.kafka.clients.consumer.ConsumerPartitionAssignor, org.apache.kafka.common.Configurable

The LagBasedPartitionAssignor operates on a per-topic basis, and attempts to assign partitions such that lag is distributed evenly across a consumer group. For each topic, we first obtain the lag on all partitions. Lag on a given partition is the difference between the end offset and the last offset committed by the consumer group. If no offsets have been committed for a partition we determine the lag based on the auto.offset.reset property. If auto.offset.reset=latest, we assign a lag of 0. If auto.offset.reset=earliest (or any other value) we assume assign lag equal to the total number of message currently available in that partition. We then create a map storing the current total lag of all partitions assigned to each member of the consumer group. Partitions are assigned in decreasing order of lag, with each partition assigned to the consumer with least total number of assigned partitions, breaking ties by assigning to the consumer with the least total assigned lag. Distributing partitions as evenly across consumers (by count) ensures that the partition assignment is balanced when all partitions have a current lag of 0 or if the distribution of lags is heavily skewed. It also gives the consumer group the best possible chance of remaining balanced if the assignment is retained for a long period. For example, suppose there are two consumers C0 and C1, both subscribed to a topic t0 having 3 partitions with the following lags: t0p0: 100,000 t0p1: 50,000 t0p2: 60,000 The assignment will be: C0: [t0p0] C1: [t0p1, t0p2] The total lag or partitions assigned to each consumer will be: C0: 100,000 C1: 110,000 Compare this to the assignments made by the RangeAssignor: C0: [t0p0, t0p1] C1: [t0p2] The RangeAssignor results in a less balanced total lag for each consumer of: C0: 160,000 C1: 50,000

See Also:

ConsumerPartitionAssignor, AbstractPartitionAssignor, RangeAssignor

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.kafka.clients.consumer.ConsumerPartitionAssignor

org.apache.kafka.clients.consumer.ConsumerPartitionAssignor.Assignment, org.apache.kafka.clients.consumer.ConsumerPartitionAssignor.GroupAssignment, org.apache.kafka.clients.consumer.ConsumerPartitionAssignor.GroupSubscription, org.apache.kafka.clients.consumer.ConsumerPartitionAssignor.RebalanceProtocol, org.apache.kafka.clients.consumer.ConsumerPartitionAssignor.Subscription

Constructor Summary

Constructors

Constructor and Description
LagBasedPartitionAssignor()

Method Summary

Modifier and Type	Method and Description
org.apache.kafka.clients.consumer.ConsumerPartitionAssignor.GroupAssignment	assign(org.apache.kafka.common.Cluster metadata, org.apache.kafka.clients.consumer.ConsumerPartitionAssignor.GroupSubscription subscriptions)
void	configure(Map<String,?> configs) This class is instantiated by reflection, at which point Kafka passes in the consumer config properties for the current instance via this method.
String	name()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.kafka.clients.consumer.ConsumerPartitionAssignor

onAssignment, subscriptionUserData, supportedProtocols, version

Constructor Detail

LagBasedPartitionAssignor

public LagBasedPartitionAssignor()

Method Detail

configure

public void configure(Map<String,?> configs)

This class is instantiated by reflection, at which point Kafka passes in the consumer config properties for the current instance via this method.

Specified by:

configure in interface org.apache.kafka.common.Configurable

Parameters:

configs - Kafka consumer configuration properties for the current instance

name

public String name()

Specified by:

name in interface org.apache.kafka.clients.consumer.ConsumerPartitionAssignor

assign

public org.apache.kafka.clients.consumer.ConsumerPartitionAssignor.GroupAssignment assign(org.apache.kafka.common.Cluster metadata,
                                                                                          org.apache.kafka.clients.consumer.ConsumerPartitionAssignor.GroupSubscription subscriptions)

Specified by:

assign in interface org.apache.kafka.clients.consumer.ConsumerPartitionAssignor