io.debezium.connector.base

Class ChangeEventQueue<T>

java.lang.Object

io.debezium.connector.base.ChangeEventQueue<T>

Type Parameters:

T - the type of events in this queue. Usually SourceRecord is used, but in cases where additional metadata must be passed from producers to the consumer, a custom type wrapping source records may be used.

All Implemented Interfaces:

ChangeEventQueueMetrics

@ThreadSafe
public class ChangeEventQueue<T>
extends Object
implements ChangeEventQueueMetrics

A queue which serves as handover point between producer threads (e.g. MySQL's binlog reader thread) and the Kafka Connect polling loop.

The queue is configurable in different aspects, e.g. its maximum size and the time to sleep (block) between two subsequent poll calls. See the ChangeEventQueue.Builder for the different options. The queue applies back-pressure semantics, i.e. if it holds the maximum number of elements, subsequent calls to enqueue(Object) will block until elements have been removed from the queue.

If an exception occurs on the producer side, the producer should make that exception known by calling producerException(RuntimeException) before stopping its operation. Upon the next call to poll(), that exception will be raised, causing Kafka Connect to stop the connector and mark it as FAILED.

Author:

Gunnar Morling

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ChangeEventQueue.Builder<T>

Field Summary

Fields

Modifier and Type	Field and Description
private T	bufferedEvent
private boolean	buffering
private AtomicLong	currentQueueSizeInBytes
private static org.slf4j.Logger	LOGGER
private Supplier<LoggingContext.PreviousContext>	loggingContextSupplier
private int	maxBatchSize
private int	maxQueueSize
private long	maxQueueSizeInBytes
private Metronome	metronome
private Map<T,Long>	objectMap
private Duration	pollInterval
private RuntimeException	producerException
private BlockingQueue<T>	queue

Constructor Summary

Constructors

Modifier	Constructor and Description
private	ChangeEventQueue(Duration pollInterval, int maxQueueSize, int maxBatchSize, Supplier<LoggingContext.PreviousContext> loggingContextSupplier, long maxQueueSizeInBytes, boolean buffering)

Method Summary

Modifier and Type	Method and Description
long	currentQueueSizeInBytes()
void	disableBuffering() Disable buffering for the queue
protected void	doEnqueue(T record)
void	enqueue(T record) Enqueues a record so that it can be obtained via poll().
void	flushBuffer(Function<T,T> recordModifier) Applies a function to the event and the buffer and adds it to the queue.
long	maxQueueSizeInBytes()
List<T>	poll() Returns the next batch of elements from this queue.
void	producerException(RuntimeException producerException)
int	remainingCapacity()
private void	throwProducerExceptionIfPresent()
int	totalCapacity()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOGGER

private static final org.slf4j.Logger LOGGER

pollInterval

private final Duration pollInterval

maxBatchSize

private final int maxBatchSize

maxQueueSize

private final int maxQueueSize

maxQueueSizeInBytes

private final long maxQueueSizeInBytes

queue

private final BlockingQueue<T> queue

metronome

private final Metronome metronome

loggingContextSupplier

private final Supplier<LoggingContext.PreviousContext> loggingContextSupplier

currentQueueSizeInBytes

private final AtomicLong currentQueueSizeInBytes

objectMap

private final Map<T,Long> objectMap

buffering

private boolean buffering

bufferedEvent

private T bufferedEvent

producerException

private volatile RuntimeException producerException

Constructor Detail

ChangeEventQueue

private ChangeEventQueue(Duration pollInterval,
                         int maxQueueSize,
                         int maxBatchSize,
                         Supplier<LoggingContext.PreviousContext> loggingContextSupplier,
                         long maxQueueSizeInBytes,
                         boolean buffering)

Method Detail

enqueue

public void enqueue(T record)
             throws InterruptedException

Enqueues a record so that it can be obtained via poll(). This method will block if the queue is full.

Parameters:

record - the record to be enqueued

Throws:

InterruptedException - if this thread has been interrupted

flushBuffer

public void flushBuffer(Function<T,T> recordModifier)
                 throws InterruptedException

Applies a function to the event and the buffer and adds it to the queue. Buffer is emptied.

Parameters:

recordModifier -

Throws:

InterruptedException

disableBuffering

public void disableBuffering()

Disable buffering for the queue

doEnqueue

protected void doEnqueue(T record)
                  throws InterruptedException

Throws:

InterruptedException

poll

public List<T> poll()
             throws InterruptedException

Returns the next batch of elements from this queue. May be empty in case no elements have arrived in the maximum waiting time.

Throws:

InterruptedException - if this thread has been interrupted while waiting for more elements to arrive

producerException

public void producerException(RuntimeException producerException)

throwProducerExceptionIfPresent

private void throwProducerExceptionIfPresent()

totalCapacity

public int totalCapacity()

Specified by:

totalCapacity in interface ChangeEventQueueMetrics

remainingCapacity

public int remainingCapacity()

Specified by:

remainingCapacity in interface ChangeEventQueueMetrics

maxQueueSizeInBytes

public long maxQueueSizeInBytes()

Specified by:

maxQueueSizeInBytes in interface ChangeEventQueueMetrics

currentQueueSizeInBytes

public long currentQueueSizeInBytes()

Specified by:

currentQueueSizeInBytes in interface ChangeEventQueueMetrics