io.druid.indexing.kafka.supervisor

Class KafkaSupervisor

java.lang.Object

io.druid.indexing.kafka.supervisor.KafkaSupervisor

All Implemented Interfaces:

Supervisor

public class KafkaSupervisor
extends Object
implements Supervisor

Supervisor responsible for managing the KafkaIndexTasks for a single dataSource. At a high level, the class accepts a KafkaSupervisorSpec which includes the Kafka topic and configuration as well as an ingestion spec which will be used to generate the indexing tasks. The run loop periodically refreshes its view of the Kafka topic's partitions and the list of running indexing tasks and ensures that all partitions are being read from and that there are enough tasks to satisfy the desired number of replicas. As tasks complete, new tasks are queued to process the next range of Kafka offsets.

Field Summary

Fields

Modifier and Type	Field and Description
static String	IS_INCREMENTAL_HANDOFF_SUPPORTED

Constructor Summary

Constructors

Constructor and Description
KafkaSupervisor(TaskStorage taskStorage, TaskMaster taskMaster, IndexerMetadataStorageCoordinator indexerMetadataStorageCoordinator, KafkaIndexTaskClientFactory taskClientFactory, com.fasterxml.jackson.databind.ObjectMapper mapper, KafkaSupervisorSpec spec)

Method Summary

Modifier and Type	Method and Description
void	checkpoint(String sequenceName, DataSourceMetadata previousCheckpoint, DataSourceMetadata currentCheckpoint)
SupervisorReport	getStatus()
protected int	getTaskGroupIdForPartition(int partition)
void	possiblyRegisterListener()
void	reset(DataSourceMetadata dataSourceMetadata)
void	start()
void	stop(boolean stopGracefully)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

IS_INCREMENTAL_HANDOFF_SUPPORTED

public static final String IS_INCREMENTAL_HANDOFF_SUPPORTED

See Also:

Constant Field Values

Constructor Detail

KafkaSupervisor

public KafkaSupervisor(TaskStorage taskStorage,
                       TaskMaster taskMaster,
                       IndexerMetadataStorageCoordinator indexerMetadataStorageCoordinator,
                       KafkaIndexTaskClientFactory taskClientFactory,
                       com.fasterxml.jackson.databind.ObjectMapper mapper,
                       KafkaSupervisorSpec spec)

Method Detail

start

public void start()

Specified by:

start in interface Supervisor

stop

public void stop(boolean stopGracefully)

Specified by:

stop in interface Supervisor

getStatus

public SupervisorReport getStatus()

Specified by:

getStatus in interface Supervisor

reset

public void reset(DataSourceMetadata dataSourceMetadata)

Specified by:

reset in interface Supervisor

checkpoint

public void checkpoint(String sequenceName,
                       DataSourceMetadata previousCheckpoint,
                       DataSourceMetadata currentCheckpoint)

Specified by:

checkpoint in interface Supervisor

possiblyRegisterListener

public void possiblyRegisterListener()

getTaskGroupIdForPartition

protected int getTaskGroupIdForPartition(int partition)