io.debezium.connector.postgresql

Class PostgresStreamingChangeEventSource

java.lang.Object

io.debezium.connector.postgresql.PostgresStreamingChangeEventSource

All Implemented Interfaces:

ChangeEventSource, StreamingChangeEventSource

public class PostgresStreamingChangeEventSource
extends Object
implements StreamingChangeEventSource

Author:

Horia Chiorean (hchiorea@redhat.com), Jiri Pechanec

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	PostgresStreamingChangeEventSource.PgConnectionSupplier

Nested classes/interfaces inherited from interface io.debezium.pipeline.source.spi.ChangeEventSource

ChangeEventSource.ChangeEventSourceContext

Field Summary

Fields

Modifier and Type	Field and Description
private Clock	clock
private PostgresConnection	connection
private PostgresConnectorConfig	connectorConfig
private EventDispatcher<TableId>	dispatcher
private ErrorHandler	errorHandler
private static int	GROWING_WAL_WARNING_LOG_INTERVAL Number of received events without sending anything to Kafka which will trigger a "WAL backlog growing" warning.
private Long	lastCompletelyProcessedLsn
private static org.slf4j.Logger	LOGGER
private long	numberOfEventsSinceLastEventSentOrWalGrowingWarning The minimum of (number of event received since the last event sent to Kafka, number of event received since last WAL growing warning issued).
private PostgresOffsetContext	offsetContext
private Metronome	pauseNoMessage
private ReplicationConnection	replicationConnection
private AtomicReference<ReplicationStream>	replicationStream
private PostgresSchema	schema
private Snapshotter	snapshotter
private PostgresTaskContext	taskContext
private static int	THROTTLE_NO_MESSAGE_BEFORE_PAUSE

Constructor Summary

Constructors

Constructor and Description
PostgresStreamingChangeEventSource(PostgresConnectorConfig connectorConfig, Snapshotter snapshotter, PostgresOffsetContext offsetContext, PostgresConnection connection, EventDispatcher<TableId> dispatcher, ErrorHandler errorHandler, Clock clock, PostgresSchema schema, PostgresTaskContext taskContext, ReplicationConnection replicationConnection)

Method Summary

Modifier and Type	Method and Description
void	commitOffset(Map<String,?> offset)
void	execute(ChangeEventSource.ChangeEventSourceContext context)
private void	maybeWarnAboutGrowingWalBacklog(boolean dispatched) If we receive change events but all of them get filtered out, we cannot commit any new offset with Apache Kafka.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

GROWING_WAL_WARNING_LOG_INTERVAL

private static final int GROWING_WAL_WARNING_LOG_INTERVAL

Number of received events without sending anything to Kafka which will trigger a "WAL backlog growing" warning.

See Also:

Constant Field Values

LOGGER

private static final org.slf4j.Logger LOGGER

THROTTLE_NO_MESSAGE_BEFORE_PAUSE

private static final int THROTTLE_NO_MESSAGE_BEFORE_PAUSE

See Also:

Constant Field Values

connection

private final PostgresConnection connection

dispatcher

private final EventDispatcher<TableId> dispatcher

errorHandler

private final ErrorHandler errorHandler

clock

private final Clock clock

schema

private final PostgresSchema schema

offsetContext

private final PostgresOffsetContext offsetContext

connectorConfig

private final PostgresConnectorConfig connectorConfig

taskContext

private final PostgresTaskContext taskContext

replicationConnection

private final ReplicationConnection replicationConnection

replicationStream

private final AtomicReference<ReplicationStream> replicationStream

snapshotter

private final Snapshotter snapshotter

pauseNoMessage

private final Metronome pauseNoMessage

numberOfEventsSinceLastEventSentOrWalGrowingWarning

private long numberOfEventsSinceLastEventSentOrWalGrowingWarning

The minimum of (number of event received since the last event sent to Kafka, number of event received since last WAL growing warning issued).

lastCompletelyProcessedLsn

private Long lastCompletelyProcessedLsn

Constructor Detail

PostgresStreamingChangeEventSource

public PostgresStreamingChangeEventSource(PostgresConnectorConfig connectorConfig,
                                          Snapshotter snapshotter,
                                          PostgresOffsetContext offsetContext,
                                          PostgresConnection connection,
                                          EventDispatcher<TableId> dispatcher,
                                          ErrorHandler errorHandler,
                                          Clock clock,
                                          PostgresSchema schema,
                                          PostgresTaskContext taskContext,
                                          ReplicationConnection replicationConnection)

Method Detail

execute

public void execute(ChangeEventSource.ChangeEventSourceContext context)
             throws InterruptedException

Specified by:

execute in interface StreamingChangeEventSource

Throws:

InterruptedException

maybeWarnAboutGrowingWalBacklog

private void maybeWarnAboutGrowingWalBacklog(boolean dispatched)

If we receive change events but all of them get filtered out, we cannot commit any new offset with Apache Kafka. This in turn means no LSN is ever acknowledged with the replication slot, causing any ever growing WAL backlog.

This situation typically occurs if there are changes on the database server, (e.g. in a blacklisted database), but none of them is in a whitelisted table. To prevent this, heartbeats can be used, as they will allow us to commit offsets also when not propagating any "real" change event.

The purpose of this method is to detect this situation and log a warning every GROWING_WAL_WARNING_LOG_INTERVAL filtered events.

Parameters:

dispatched - Whether an event was sent to the broker or not

commitOffset

public void commitOffset(Map<String,?> offset)

Specified by:

commitOffset in interface StreamingChangeEventSource