Package io.debezium.connector.postgresql

Class PostgresStreamingChangeEventSource

java.lang.Object

io.debezium.connector.postgresql.PostgresStreamingChangeEventSource

All Implemented Interfaces:

ChangeEventSource, StreamingChangeEventSource<PostgresPartition,PostgresOffsetContext>

public class PostgresStreamingChangeEventSource
extends Object
implements StreamingChangeEventSource<PostgresPartition,PostgresOffsetContext>

Author:

Horia Chiorean (hchiorea@redhat.com), Jiri Pechanec

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static interface	PostgresStreamingChangeEventSource.PgConnectionSupplier

Nested classes/interfaces inherited from interface io.debezium.pipeline.source.spi.ChangeEventSource

ChangeEventSource.ChangeEventSourceContext

Field Summary

Fields

Modifier and Type	Field	Description
private final Clock	clock
private final PostgresConnection	connection
private final ElapsedTimeStrategy	connectionProbeTimer
private final PostgresConnectorConfig	connectorConfig
private final PostgresEventDispatcher<TableId>	dispatcher
private PostgresOffsetContext	effectiveOffset
private final ErrorHandler	errorHandler
private static final int	GROWING_WAL_WARNING_LOG_INTERVAL	Number of received events without sending anything to Kafka which will trigger a "WAL backlog growing" warning.
private static final String	KEEP_ALIVE_THREAD_NAME
private Lsn	lastCompletelyProcessedLsn
private static final org.slf4j.Logger	LOGGER
private boolean	lsnFlushingAllowed
private long	numberOfEventsSinceLastEventSentOrWalGrowingWarning	The minimum of (number of event received since the last event sent to Kafka, number of event received since last WAL growing warning issued).
private final DelayStrategy	pauseNoMessage
private final ReplicationConnection	replicationConnection
private final AtomicReference<ReplicationStream>	replicationStream
private final PostgresSchema	schema
private final SnapshotterService	snapshotterService
private final PostgresTaskContext	taskContext
private static final int	THROTTLE_NO_MESSAGE_BEFORE_PAUSE

Constructor Summary

Constructors

Constructor	Description
PostgresStreamingChangeEventSource(PostgresConnectorConfig connectorConfig, SnapshotterService snapshotterService, PostgresConnection connection, PostgresEventDispatcher<TableId> dispatcher, ErrorHandler errorHandler, Clock clock, PostgresSchema schema, PostgresTaskContext taskContext, ReplicationConnection replicationConnection)

Method Summary

Modifier and Type	Method	Description
private void	commitMessage(PostgresPartition partition, PostgresOffsetContext offsetContext, Lsn lsn)
void	commitOffset(Map<String,?> partition, Map<String,?> offset)
void	execute(ChangeEventSource.ChangeEventSourceContext context, PostgresPartition partition, PostgresOffsetContext offsetContext)
PostgresOffsetContext	getOffsetContext()
void	init(PostgresOffsetContext offsetContext)
private void	initSchema()
private boolean	isInPreSnapshotCatchUpStreaming(PostgresOffsetContext offsetContext)	Returns whether the current streaming phase is running a catch up streaming phase that runs before a snapshot.
private void	maybeWarnAboutGrowingWalBacklog(boolean dispatched)	If we receive change events but all of them get filtered out, we cannot commit any new offset with Apache Kafka.
private void	probeConnectionIfNeeded()
private void	processMessages(ChangeEventSource.ChangeEventSourceContext context, PostgresPartition partition, PostgresOffsetContext offsetContext, ReplicationStream stream)
private void	processReplicationMessages(PostgresPartition partition, PostgresOffsetContext offsetContext, ReplicationStream stream, ReplicationMessage message)
private void	searchWalPosition(ChangeEventSource.ChangeEventSourceContext context, PostgresPartition partition, PostgresOffsetContext offsetContext, ReplicationStream stream, WalPositionLocator walPosition)
private Long	toLong(OptionalLong l)
private String	toString(OptionalLong l)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.debezium.pipeline.source.spi.StreamingChangeEventSource

executeIteration

Field Details

KEEP_ALIVE_THREAD_NAME

private static final String KEEP_ALIVE_THREAD_NAME

See Also:

• Constant Field Values

GROWING_WAL_WARNING_LOG_INTERVAL

private static final int GROWING_WAL_WARNING_LOG_INTERVAL

Number of received events without sending anything to Kafka which will trigger a "WAL backlog growing" warning.

See Also:

• Constant Field Values

LOGGER

private static final org.slf4j.Logger LOGGER

THROTTLE_NO_MESSAGE_BEFORE_PAUSE

private static final int THROTTLE_NO_MESSAGE_BEFORE_PAUSE

See Also:

• Constant Field Values

connection

private final PostgresConnection connection

dispatcher

private final PostgresEventDispatcher<TableId> dispatcher

errorHandler

private final ErrorHandler errorHandler

clock

private final Clock clock

schema

private final PostgresSchema schema

connectorConfig

private final PostgresConnectorConfig connectorConfig

taskContext

private final PostgresTaskContext taskContext

replicationConnection

private final ReplicationConnection replicationConnection

replicationStream

private final AtomicReference<ReplicationStream> replicationStream

snapshotterService

private final SnapshotterService snapshotterService

pauseNoMessage

private final DelayStrategy pauseNoMessage

connectionProbeTimer

private final ElapsedTimeStrategy connectionProbeTimer

lsnFlushingAllowed

private volatile boolean lsnFlushingAllowed

numberOfEventsSinceLastEventSentOrWalGrowingWarning

private long numberOfEventsSinceLastEventSentOrWalGrowingWarning

The minimum of (number of event received since the last event sent to Kafka, number of event received since last WAL growing warning issued).

lastCompletelyProcessedLsn

private Lsn lastCompletelyProcessedLsn

effectiveOffset

private PostgresOffsetContext effectiveOffset

Constructor Details

PostgresStreamingChangeEventSource

public PostgresStreamingChangeEventSource(PostgresConnectorConfig connectorConfig,
 SnapshotterService snapshotterService,
 PostgresConnection connection,
 PostgresEventDispatcher<TableId> dispatcher,
 ErrorHandler errorHandler,
 Clock clock,
 PostgresSchema schema,
 PostgresTaskContext taskContext,
 ReplicationConnection replicationConnection)

Method Details

init

public void init(PostgresOffsetContext offsetContext)

Specified by:

init in interface StreamingChangeEventSource<PostgresPartition,PostgresOffsetContext>

initSchema

private void initSchema()

execute

public void execute(ChangeEventSource.ChangeEventSourceContext context,
 PostgresPartition partition,
 PostgresOffsetContext offsetContext)
             throws InterruptedException

Specified by:

execute in interface StreamingChangeEventSource<PostgresPartition,PostgresOffsetContext>

Throws:

InterruptedException

processMessages

private void processMessages(ChangeEventSource.ChangeEventSourceContext context,
 PostgresPartition partition,
 PostgresOffsetContext offsetContext,
 ReplicationStream stream)
                      throws SQLException,
InterruptedException

Throws:

SQLException

InterruptedException

processReplicationMessages

private void processReplicationMessages(PostgresPartition partition,
 PostgresOffsetContext offsetContext,
 ReplicationStream stream,
 ReplicationMessage message)
                                 throws SQLException,
InterruptedException

Throws:

SQLException

InterruptedException

searchWalPosition

private void searchWalPosition(ChangeEventSource.ChangeEventSourceContext context,
 PostgresPartition partition,
 PostgresOffsetContext offsetContext,
 ReplicationStream stream,
 WalPositionLocator walPosition)
                        throws SQLException,
InterruptedException

Throws:

SQLException

InterruptedException

probeConnectionIfNeeded

private void probeConnectionIfNeeded()
                              throws SQLException

Throws:

SQLException

commitMessage

private void commitMessage(PostgresPartition partition,
 PostgresOffsetContext offsetContext,
 Lsn lsn)
                    throws SQLException,
InterruptedException

Throws:

SQLException

InterruptedException

maybeWarnAboutGrowingWalBacklog

private void maybeWarnAboutGrowingWalBacklog(boolean dispatched)

If we receive change events but all of them get filtered out, we cannot commit any new offset with Apache Kafka. This in turn means no LSN is ever acknowledged with the replication slot, causing any ever growing WAL backlog.

This situation typically occurs if there are changes on the database server, (e.g. in an excluded database), but none of them is in table.include.list. To prevent this, heartbeats can be used, as they will allow us to commit offsets also when not propagating any "real" change event.

The purpose of this method is to detect this situation and log a warning every GROWING_WAL_WARNING_LOG_INTERVAL filtered events.

Parameters:

dispatched - Whether an event was sent to the broker or not

commitOffset

public void commitOffset(Map<String,?> partition,
 Map<String,?> offset)

Specified by:

commitOffset in interface StreamingChangeEventSource<PostgresPartition,PostgresOffsetContext>

getOffsetContext

public PostgresOffsetContext getOffsetContext()

Specified by:

getOffsetContext in interface StreamingChangeEventSource<PostgresPartition,PostgresOffsetContext>

isInPreSnapshotCatchUpStreaming

private boolean isInPreSnapshotCatchUpStreaming(PostgresOffsetContext offsetContext)

Returns whether the current streaming phase is running a catch up streaming phase that runs before a snapshot. This is useful for transaction management. During pre-snapshot catch up streaming, we open the snapshot transaction early and hold the transaction open throughout the pre snapshot catch up streaming phase so that we know where to stop streaming and can start the snapshot phase at a consistent location. This is opposed the regular streaming, where we do not a lingering open transaction.

Returns:

true if the current streaming phase is performing catch up streaming

toLong

private Long toLong(OptionalLong l)

toString

private String toString(OptionalLong l)