Package io.debezium.connector.postgresql

Class PostgresStreamingChangeEventSource

java.lang.Object

io.debezium.connector.postgresql.PostgresStreamingChangeEventSource

All Implemented Interfaces:

ChangeEventSource, StreamingChangeEventSource<PostgresPartition,PostgresOffsetContext>

public class PostgresStreamingChangeEventSource
extends Object
implements StreamingChangeEventSource<PostgresPartition,PostgresOffsetContext>

Author:

Horia Chiorean (hchiorea@redhat.com), Jiri Pechanec

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static interface	PostgresStreamingChangeEventSource.PgConnectionSupplier

Nested classes/interfaces inherited from interface io.debezium.pipeline.source.spi.ChangeEventSource

ChangeEventSource.ChangeEventSourceContext

Field Summary

Fields

Modifier and Type	Field	Description
private Clock	clock
private PostgresConnection	connection
private ElapsedTimeStrategy	connectionProbeTimer
private PostgresConnectorConfig	connectorConfig
private EventDispatcher<TableId>	dispatcher
private ErrorHandler	errorHandler
private static int	GROWING_WAL_WARNING_LOG_INTERVAL	Number of received events without sending anything to Kafka which will trigger a "WAL backlog growing" warning.
private static String	KEEP_ALIVE_THREAD_NAME
private Lsn	lastCompletelyProcessedLsn
private static org.slf4j.Logger	LOGGER
private long	numberOfEventsSinceLastEventSentOrWalGrowingWarning	The minimum of (number of event received since the last event sent to Kafka, number of event received since last WAL growing warning issued).
private DelayStrategy	pauseNoMessage
private ReplicationConnection	replicationConnection
private AtomicReference<ReplicationStream>	replicationStream
private PostgresSchema	schema
private Snapshotter	snapshotter
private PostgresTaskContext	taskContext
private static int	THROTTLE_NO_MESSAGE_BEFORE_PAUSE

Constructor Summary

Constructors

Constructor	Description
PostgresStreamingChangeEventSource(PostgresConnectorConfig connectorConfig, Snapshotter snapshotter, PostgresConnection connection, EventDispatcher<TableId> dispatcher, ErrorHandler errorHandler, Clock clock, PostgresSchema schema, PostgresTaskContext taskContext, ReplicationConnection replicationConnection)

Method Summary

Modifier and Type	Method	Description
private void	commitMessage(PostgresPartition partition, PostgresOffsetContext offsetContext, Lsn lsn)
void	commitOffset(Map<String,?> offset)
void	execute(ChangeEventSource.ChangeEventSourceContext context, PostgresPartition partition, PostgresOffsetContext offsetContext)
void	init()
private boolean	isInPreSnapshotCatchUpStreaming(PostgresOffsetContext offsetContext)	Returns whether the current streaming phase is running a catch up streaming phase that runs before a snapshot.
private void	maybeWarnAboutGrowingWalBacklog(boolean dispatched)	If we receive change events but all of them get filtered out, we cannot commit any new offset with Apache Kafka.
private void	probeConnectionIfNeeded()
private void	processMessages(ChangeEventSource.ChangeEventSourceContext context, PostgresPartition partition, PostgresOffsetContext offsetContext, ReplicationStream stream)
private void	searchWalPosition(ChangeEventSource.ChangeEventSourceContext context, ReplicationStream stream, WalPositionLocator walPosition)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

KEEP_ALIVE_THREAD_NAME

private static final String KEEP_ALIVE_THREAD_NAME

See Also:

Constant Field Values

GROWING_WAL_WARNING_LOG_INTERVAL

private static final int GROWING_WAL_WARNING_LOG_INTERVAL

Number of received events without sending anything to Kafka which will trigger a "WAL backlog growing" warning.

See Also:

Constant Field Values

LOGGER

private static final org.slf4j.Logger LOGGER

THROTTLE_NO_MESSAGE_BEFORE_PAUSE

private static final int THROTTLE_NO_MESSAGE_BEFORE_PAUSE

See Also:

Constant Field Values

connection

private final PostgresConnection connection

dispatcher

private final EventDispatcher<TableId> dispatcher

errorHandler

private final ErrorHandler errorHandler

clock

private final Clock clock

schema

private final PostgresSchema schema

connectorConfig

private final PostgresConnectorConfig connectorConfig

taskContext

private final PostgresTaskContext taskContext

replicationConnection

private final ReplicationConnection replicationConnection

replicationStream

private final AtomicReference<ReplicationStream> replicationStream

snapshotter

private final Snapshotter snapshotter

pauseNoMessage

private final DelayStrategy pauseNoMessage

connectionProbeTimer

private final ElapsedTimeStrategy connectionProbeTimer

numberOfEventsSinceLastEventSentOrWalGrowingWarning

private long numberOfEventsSinceLastEventSentOrWalGrowingWarning

The minimum of (number of event received since the last event sent to Kafka, number of event received since last WAL growing warning issued).

lastCompletelyProcessedLsn

private Lsn lastCompletelyProcessedLsn

Constructor Detail

PostgresStreamingChangeEventSource

public PostgresStreamingChangeEventSource(PostgresConnectorConfig connectorConfig,
                                          Snapshotter snapshotter,
                                          PostgresConnection connection,
                                          EventDispatcher<TableId> dispatcher,
                                          ErrorHandler errorHandler,
                                          Clock clock,
                                          PostgresSchema schema,
                                          PostgresTaskContext taskContext,
                                          ReplicationConnection replicationConnection)

Method Detail

init

public void init()

Specified by:

init in interface StreamingChangeEventSource<PostgresPartition,PostgresOffsetContext>

execute

public void execute(ChangeEventSource.ChangeEventSourceContext context,
                    PostgresPartition partition,
                    PostgresOffsetContext offsetContext)
             throws InterruptedException

Specified by:

execute in interface StreamingChangeEventSource<PostgresPartition,PostgresOffsetContext>

Throws:

InterruptedException

processMessages

private void processMessages(ChangeEventSource.ChangeEventSourceContext context,
                             PostgresPartition partition,
                             PostgresOffsetContext offsetContext,
                             ReplicationStream stream)
                      throws SQLException,
                             InterruptedException

Throws:

SQLException

InterruptedException

searchWalPosition

private void searchWalPosition(ChangeEventSource.ChangeEventSourceContext context,
                               ReplicationStream stream,
                               WalPositionLocator walPosition)
                        throws SQLException,
                               InterruptedException

Throws:

SQLException

InterruptedException

probeConnectionIfNeeded

private void probeConnectionIfNeeded()
                              throws SQLException

Throws:

SQLException

commitMessage

private void commitMessage(PostgresPartition partition,
                           PostgresOffsetContext offsetContext,
                           Lsn lsn)
                    throws SQLException,
                           InterruptedException

Throws:

SQLException

InterruptedException

maybeWarnAboutGrowingWalBacklog

private void maybeWarnAboutGrowingWalBacklog(boolean dispatched)

If we receive change events but all of them get filtered out, we cannot commit any new offset with Apache Kafka. This in turn means no LSN is ever acknowledged with the replication slot, causing any ever growing WAL backlog.

This situation typically occurs if there are changes on the database server, (e.g. in an excluded database), but none of them is in table.include.list. To prevent this, heartbeats can be used, as they will allow us to commit offsets also when not propagating any "real" change event.

The purpose of this method is to detect this situation and log a warning every GROWING_WAL_WARNING_LOG_INTERVAL filtered events.

Parameters:

dispatched - Whether an event was sent to the broker or not

commitOffset

public void commitOffset(Map<String,?> offset)

Specified by:

commitOffset in interface StreamingChangeEventSource<PostgresPartition,PostgresOffsetContext>

isInPreSnapshotCatchUpStreaming

private boolean isInPreSnapshotCatchUpStreaming(PostgresOffsetContext offsetContext)

Returns whether the current streaming phase is running a catch up streaming phase that runs before a snapshot. This is useful for transaction management. During pre-snapshot catch up streaming, we open the snapshot transaction early and hold the transaction open throughout the pre snapshot catch up streaming phase so that we know where to stop streaming and can start the snapshot phase at a consistent location. This is opposed the regular streaming, where we do not a lingering open transaction.

Returns:

true if the current streaming phase is performing catch up streaming