Package io.debezium.connector.postgresql

Class PostgresReadOnlyIncrementalSnapshotChangeEventSource<P extends PostgresPartition>

java.lang.Object

io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource<P,TableId>

io.debezium.connector.postgresql.PostgresReadOnlyIncrementalSnapshotChangeEventSource<P>

All Implemented Interfaces:

IncrementalSnapshotChangeEventSource<P,TableId>

public class PostgresReadOnlyIncrementalSnapshotChangeEventSource<P extends PostgresPartition>
extends AbstractIncrementalSnapshotChangeEventSource<P,TableId>

A PostgreSQL specific read-only incremental snapshot change event source. Uses PgSnapshot as low/high watermarks for incremental snapshot window.

Prerequisites

• PostgreSQL version >= 13

How a chunk is snapshotted

• a pg_current_snapshot() query is executed and the low watermark is set to lowWatermark
• a new data chunk is read from a database by generating the SELECT statement and placed into a window buffer keyed by primary keys
• a pg_current_snapshot() query is executed and the high watermark is set to highWatermark

During the subsequent streaming

• if WAL event is received and its txId is greater then or equal to xMin of the low watermark, then window processing mode is enabled
• if WAL event is received and its txId is greater then the xMax of high watermark, then window processing mode is disabled and the rest of the window’s buffer is streamed
• if window processing mode is enabled then if the event key is contained in the window buffer then it is removed from the window buffer
• event is streamed

No binlog events

Heartbeat events (events that are sent when there are not changes from the WAL) are used to update the window processing mode when the rate of WAL updates is low.

The heartbeat has the same txId as the latest WAL event at the moment and it will used to continue to read chunks even there are not event coming from the WAL. This processing will end if the watermarks changes because that means there are some transaction potentially modifying data.

No changes between watermarks

A window can be opened and closed right away by the same event. This can happen when a high watermark and low watermark are the same, which means there were no active transaction during the chunk select. Chunk will get inserted right after the low watermark, no events will be deduplicated from the chunk

Author:

Mario Fiore Vitale

Field Summary

Fields

Modifier and Type	Field	Description
private static final String	CURRENT_SNAPSHOT
private static final String	FORCE_NEW_TRANSACTION
private final PostgresConnection	jdbcConnection
private static final org.slf4j.Logger	LOGGER
private final PostgresSchema	schema

Fields inherited from class io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource

chunkQueryBuilder, connectorConfig, context, dispatcher, notificationService, window

Constructor Summary

Constructors

Constructor	Description
PostgresReadOnlyIncrementalSnapshotChangeEventSource(RelationalDatabaseConnectorConfig config, JdbcConnection jdbcConnection, EventDispatcher<P,TableId> dispatcher, DatabaseSchema<?> databaseSchema, Clock clock, SnapshotProgressListener<P> progressListener, DataChangeEventListener<P> dataChangeEventListener, NotificationService<P,? extends OffsetContext> notificationService)

Method Summary

Modifier and Type	Method	Description
protected void	emitWindowClose(P partition, OffsetContext offsetContext)
protected void	emitWindowOpen()
private void	forceNewTransactionId()
private PostgresReadOnlyIncrementalSnapshotContext<TableId>	getContext()
private void	getCurrentSnapshot(Consumer<PgSnapshot> watermark)
private boolean	maxInProgressTransactionCommitted(Long eventTxId)
protected void	preIncrementalSnapshotStart()
void	processHeartbeat(P partition, OffsetContext offsetContext)
void	processMessage(P partition, io.debezium.spi.schema.DataCollectionId dataCollectionId, Object key, OffsetContext offsetContext)
void	processTransactionCommittedEvent(P partition, OffsetContext offsetContext)
private void	readUntilNewTransactionChange(P partition, OffsetContext offsetContext)
protected Table	refreshTableSchema(Table table)

Methods inherited from class io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource

addDataCollectionNamesToSnapshot, closeWindow, deduplicateWindow, getChangeRecordEmitter, getSignalTableName, init, pauseSnapshot, postIncrementalSnapshotCompleted, postReadChunk, preReadChunk, processSchemaChange, readChunk, rereadChunk, resumeSnapshot, sendEvent, sendWindowEvents, setContext, stopSnapshot

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.debezium.pipeline.source.snapshot.incremental.IncrementalSnapshotChangeEventSource

processFilteredEvent, processTransactionStartedEvent

Field Details

LOGGER

private static final org.slf4j.Logger LOGGER

FORCE_NEW_TRANSACTION

private static final String FORCE_NEW_TRANSACTION

See Also:

• Constant Field Values

CURRENT_SNAPSHOT

private static final String CURRENT_SNAPSHOT

See Also:

• Constant Field Values

jdbcConnection

private final PostgresConnection jdbcConnection

schema

private final PostgresSchema schema

Constructor Details

PostgresReadOnlyIncrementalSnapshotChangeEventSource

public PostgresReadOnlyIncrementalSnapshotChangeEventSource(RelationalDatabaseConnectorConfig config,
 JdbcConnection jdbcConnection,
 EventDispatcher<P,TableId> dispatcher,
 DatabaseSchema<?> databaseSchema,
 Clock clock,
 SnapshotProgressListener<P> progressListener,
 DataChangeEventListener<P> dataChangeEventListener,
 NotificationService<P,? extends OffsetContext> notificationService)

Method Details

preIncrementalSnapshotStart

protected void preIncrementalSnapshotStart()

Overrides:

preIncrementalSnapshotStart in class AbstractIncrementalSnapshotChangeEventSource<P extends PostgresPartition,TableId>

getContext

private PostgresReadOnlyIncrementalSnapshotContext<TableId> getContext()

emitWindowOpen

protected void emitWindowOpen()

Specified by:

emitWindowOpen in class AbstractIncrementalSnapshotChangeEventSource<P extends PostgresPartition,TableId>

emitWindowClose

protected void emitWindowClose(P partition,
 OffsetContext offsetContext)

Specified by:

emitWindowClose in class AbstractIncrementalSnapshotChangeEventSource<P extends PostgresPartition,TableId>

processMessage

public void processMessage(P partition,
 io.debezium.spi.schema.DataCollectionId dataCollectionId,
 Object key,
 OffsetContext offsetContext)
                    throws InterruptedException

Throws:

InterruptedException

processTransactionCommittedEvent

public void processTransactionCommittedEvent(P partition,
 OffsetContext offsetContext)
                                      throws InterruptedException

Throws:

InterruptedException

processHeartbeat

public void processHeartbeat(P partition,
 OffsetContext offsetContext)
                      throws InterruptedException

Throws:

InterruptedException

refreshTableSchema

protected Table refreshTableSchema(Table table)
                            throws SQLException

Overrides:

refreshTableSchema in class AbstractIncrementalSnapshotChangeEventSource<P extends PostgresPartition,TableId>

Throws:

SQLException

readUntilNewTransactionChange

private void readUntilNewTransactionChange(P partition,
 OffsetContext offsetContext)
                                    throws InterruptedException

Throws:

InterruptedException

getCurrentSnapshot

private void getCurrentSnapshot(Consumer<PgSnapshot> watermark)

maxInProgressTransactionCommitted

private boolean maxInProgressTransactionCommitted(Long eventTxId)

forceNewTransactionId

private void forceNewTransactionId()