Package io.debezium.connector.postgresql.connection

Class PostgresReplicationConnection

java.lang.Object

io.debezium.jdbc.JdbcConnection

io.debezium.connector.postgresql.connection.PostgresReplicationConnection

All Implemented Interfaces:

ReplicationConnection, AutoCloseable

public class PostgresReplicationConnection
extends JdbcConnection
implements ReplicationConnection

Implementation of a ReplicationConnection for Postgresql. Note that replication connections in PG cannot execute regular statements but only a limited number of replication-related commands.

Author:

Horia Chiorean (hchiorea@redhat.com)

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
protected static class	PostgresReplicationConnection.ReplicationConnectionBuilder

Nested classes/interfaces inherited from class io.debezium.jdbc.JdbcConnection

JdbcConnection.BlockingMultiResultSetConsumer, JdbcConnection.BlockingResultSetConsumer, JdbcConnection.CallPreparer, JdbcConnection.ConnectionFactory, JdbcConnection.MultiResultSetConsumer, JdbcConnection.Operations, JdbcConnection.ParameterResultSetConsumer, JdbcConnection.ResultSetConsumer, JdbcConnection.ResultSetExtractor<T extends Object>, JdbcConnection.ResultSetMapper<T extends Object>, JdbcConnection.StatementFactory, JdbcConnection.StatementPreparer

Nested classes/interfaces inherited from interface io.debezium.connector.postgresql.connection.ReplicationConnection

ReplicationConnection.Builder

Field Summary

Fields

Modifier and Type	Field	Description
private final PostgresConnectorConfig	connectorConfig
private Lsn	defaultStartingPos
private final boolean	dropSlotOnClose
private boolean	hasInitedSlot
private final PostgresConnection	jdbcConnection
private static org.slf4j.Logger	LOGGER
private final MessageDecoder	messageDecoder
private final PostgresConnectorConfig.LogicalDecoder	plugin
private final PostgresConnectorConfig.AutoCreateMode	publicationAutocreateMode
private final String	publicationName
private Optional<ReplicaIdentityMapper>	replicaIdentityMapper
private SlotCreationResult	slotCreationInfo
private final String	slotName
private static final String	SQL_STATE_INSUFFICIENT_PRIVILEGE
private final Duration	statusUpdateInterval
private final Properties	streamParams
private final RelationalTableFilters	tableFilter
private final TypeRegistry	typeRegistry

Constructor Summary

Constructors

Modifier	Constructor	Description
private	PostgresReplicationConnection(PostgresConnectorConfig config, String slotName, String publicationName, RelationalTableFilters tableFilter, PostgresConnectorConfig.AutoCreateMode publicationAutocreateMode, PostgresConnectorConfig.LogicalDecoder plugin, boolean dropSlotOnClose, Duration statusUpdateInterval, PostgresConnection jdbcConnection, TypeRegistry typeRegistry, Properties streamParams, PostgresSchema schema)	Creates a new replication connection with the given params.

Method Summary

Modifier and Type	Method	Description
private static JdbcConfiguration	addDefaultSettings(JdbcConfiguration configuration)
void	close()
void	close(boolean dropSlot)
private void	createOrUpdatePublicationModeFilterted(Statement stmt, boolean isUpdate)
Optional<SlotCreationResult>	createReplicationSlot()	Creates a new replication slot with the given option and returns the result of the command, which may contain results (depending on postgres versions)
private ReplicationStream	createReplicationStream(Lsn startLsn, WalPositionLocator walPosition)
private Set<TableId>	determineCapturedTables()
private ServerInfo.ReplicationSlot	getSlotInfo()
private Boolean	hasMinimumVersion(int version)
void	initConnection()	Forces the connection to be created, is called by startStreaming, but can be called manually in cases where we want the connection but not to to start streaming yet.
protected void	initPublication()
private void	initReplicaIdentity()	Check all tables captured by the connector, contained in Set<TableId> from determineCapturedTables().
protected void	initReplicationSlot()
private SlotCreationResult	parseSlotCreation(ResultSet rs)
protected org.postgresql.core.BaseConnection	pgConnection()
void	reconnect()
private org.postgresql.replication.PGReplicationStream	startPgReplicationStream(Lsn lsn, BiFunction<org.postgresql.replication.fluent.logical.ChainedLogicalStreamBuilder,Function<Integer,Boolean>,org.postgresql.replication.fluent.logical.ChainedLogicalStreamBuilder> configurator)
ReplicationStream	startStreaming(Lsn offset, WalPositionLocator walPosition)	Opens a stream for reading logical replication changes from a given LSN position.
ReplicationStream	startStreaming(WalPositionLocator walPosition)	creating a replication connection and starting to stream involves a few steps: 1.
private boolean	useTemporarySlot()
protected void	validateSlotIsInExpectedState(WalPositionLocator walPosition)

Methods inherited from class io.debezium.jdbc.JdbcConnection

buildSelectWithRowLimits, call, chunkQueryBuilder, commit, config, connect, connection, connection, connectionString, createTableId, database, escapeEscapeSequence, execute, execute, executeWithoutCommitting, getAttributeDetails, getColumnsDetails, getColumnValue, getCurrentTimestamp, getQualifiedTableName, isConnected, isNullable, isTableType, isTableUniqueIndexIncluded, isValid, loadKeyStore, nullsSortLast, overrideColumn, parseSqlStatementString, patternBasedFactory, patternBasedFactory, prepareQuery, prepareQuery, prepareQuery, prepareQuery, prepareQuery, prepareQueryAndMap, prepareQueryWithBlockingConsumer, prepareUpdate, print, print, query, query, queryAndMap, queryAndMap, querySingleValue, queryWithBlockingConsumer, quotedColumnIdString, quotedTableIdString, readAllCatalogNames, readAllSchemaNames, readAllTableNames, readPrimaryKeyNames, readPrimaryKeyOrUniqueIndexNames, readSchema, readTableColumn, readTableNames, readTablePreparedStatement, readTableStatement, readTableUniqueIndices, reselectColumns, reselectColumns, resolveCatalogName, resolveJdbcType, resolveNativeType, rollback, rowToArray, setAutoCommit, setQueryColumnValue, singleResultMapper, supportedTableTypes, tableTypes, username

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.debezium.connector.postgresql.connection.ReplicationConnection

isConnected

Field Details

SQL_STATE_INSUFFICIENT_PRIVILEGE

private static final String SQL_STATE_INSUFFICIENT_PRIVILEGE

See Also:

• Constant Field Values

LOGGER

private static org.slf4j.Logger LOGGER

slotName

private final String slotName

publicationName

private final String publicationName

tableFilter

private final RelationalTableFilters tableFilter

publicationAutocreateMode

private final PostgresConnectorConfig.AutoCreateMode publicationAutocreateMode

plugin

private final PostgresConnectorConfig.LogicalDecoder plugin

dropSlotOnClose

private final boolean dropSlotOnClose

connectorConfig

private final PostgresConnectorConfig connectorConfig

statusUpdateInterval

private final Duration statusUpdateInterval

messageDecoder

private final MessageDecoder messageDecoder

jdbcConnection

private final PostgresConnection jdbcConnection

typeRegistry

private final TypeRegistry typeRegistry

streamParams

private final Properties streamParams

defaultStartingPos

private Lsn defaultStartingPos

slotCreationInfo

private SlotCreationResult slotCreationInfo

hasInitedSlot

private boolean hasInitedSlot

replicaIdentityMapper

private Optional<ReplicaIdentityMapper> replicaIdentityMapper

Constructor Details

PostgresReplicationConnection

private PostgresReplicationConnection(PostgresConnectorConfig config,
 String slotName,
 String publicationName,
 RelationalTableFilters tableFilter,
 PostgresConnectorConfig.AutoCreateMode publicationAutocreateMode,
 PostgresConnectorConfig.LogicalDecoder plugin,
 boolean dropSlotOnClose,
 Duration statusUpdateInterval,
 PostgresConnection jdbcConnection,
 TypeRegistry typeRegistry,
 Properties streamParams,
 PostgresSchema schema)

Creates a new replication connection with the given params.

Parameters:

config - the JDBC configuration for the connection; may not be null

slotName - the name of the DB slot for logical replication; may not be null

publicationName - the name of the DB publication for logical replication; may not be null

tableFilter - the tables to watch of the DB publication for logical replication; may not be null

publicationAutocreateMode - the mode for publication autocreation; may not be null

plugin - decoder matching the server side plug-in used for streaming changes; may not be null

dropSlotOnClose - whether the replication slot should be dropped once the connection is closed

statusUpdateInterval - the interval at which the replication connection should periodically send status

jdbcConnection - general PostgreSQL JDBC connection

typeRegistry - registry with PostgreSQL types

streamParams - additional parameters to pass to the replication stream

schema - the schema; must not be null

Method Details

addDefaultSettings

private static JdbcConfiguration addDefaultSettings(JdbcConfiguration configuration)

getSlotInfo

private ServerInfo.ReplicationSlot getSlotInfo()
                                        throws SQLException,
InterruptedException

Throws:

SQLException

InterruptedException

initPublication

protected void initPublication()

createOrUpdatePublicationModeFilterted

private void createOrUpdatePublicationModeFilterted(Statement stmt,
 boolean isUpdate)

initReplicaIdentity

private void initReplicaIdentity()

Check all tables captured by the connector, contained in Set<TableId> from determineCapturedTables(). Updating Replica Identity in PostgreSQL database based on PostgresConnectorConfig.REPLICA_IDENTITY_AUTOSET_VALUES configuration parameter for each TableId

Throws:

Exception

determineCapturedTables

private Set<TableId> determineCapturedTables()
                                      throws Exception

Throws:

Exception

initReplicationSlot

protected void initReplicationSlot()
                            throws SQLException,
InterruptedException

Throws:

SQLException

InterruptedException

useTemporarySlot

private boolean useTemporarySlot()
                          throws SQLException

Throws:

SQLException

startStreaming

public ReplicationStream startStreaming(WalPositionLocator walPosition)
                                 throws SQLException,
InterruptedException

creating a replication connection and starting to stream involves a few steps: 1. we create the connection and ensure that a. the slot exists b. the slot isn't currently being used 2. we query to get our potential start position in the slot (lsn) 3. we try and start streaming, depending on our options this may fail, which can result in the connection being killed and we need to start the process over if we are using a temporary slot 4. actually start the streamer

This method takes care of all of these and this method queries for a default starting position If you know where you are starting from you should call startStreaming(Lsn, WalPositionLocator), this method delegates to that method

Specified by:

startStreaming in interface ReplicationConnection

Returns:

Throws:

SQLException

InterruptedException

startStreaming

public ReplicationStream startStreaming(Lsn offset,
 WalPositionLocator walPosition)
                                 throws SQLException,
InterruptedException

Description copied from interface: ReplicationConnection

Opens a stream for reading logical replication changes from a given LSN position.

Note that it is possible for a server to have recycled old WAL segments (see the wal_keep_segments setting). If that is the case, then even though a LSN number may be valid, the server will not stream back any changes because they are not available.

Specified by:

startStreaming in interface ReplicationConnection

Parameters:

offset - a value representing the WAL sequence number where replication should start from; if the value is null or negative, this behaves exactly like #startStreaming().

Returns:

a PGReplicationStream from which data is read; never null

Throws:

SQLException - if anything fails

InterruptedException

See Also:

• LogSequenceNumber

validateSlotIsInExpectedState

protected void validateSlotIsInExpectedState(WalPositionLocator walPosition)
                                      throws SQLException

Throws:

SQLException

initConnection

public void initConnection()
                    throws SQLException,
InterruptedException

Description copied from interface: ReplicationConnection

Forces the connection to be created, is called by startStreaming, but can be called manually in cases where we want the connection but not to to start streaming yet. Can be called multiple times

Specified by:

initConnection in interface ReplicationConnection

Throws:

SQLException

InterruptedException

createReplicationSlot

public Optional<SlotCreationResult> createReplicationSlot()
                                                   throws SQLException

Description copied from interface: ReplicationConnection

Creates a new replication slot with the given option and returns the result of the command, which may contain results (depending on postgres versions)

Specified by:

createReplicationSlot in interface ReplicationConnection

Throws:

SQLException - if anything fails

pgConnection

protected org.postgresql.core.BaseConnection pgConnection()
                                                   throws SQLException

Throws:

SQLException

parseSlotCreation

private SlotCreationResult parseSlotCreation(ResultSet rs)

createReplicationStream

private ReplicationStream createReplicationStream(Lsn startLsn,
 WalPositionLocator walPosition)
                                           throws SQLException,
InterruptedException

Throws:

SQLException

InterruptedException

startPgReplicationStream

private org.postgresql.replication.PGReplicationStream startPgReplicationStream(Lsn lsn,
 BiFunction<org.postgresql.replication.fluent.logical.ChainedLogicalStreamBuilder,Function<Integer,Boolean>,org.postgresql.replication.fluent.logical.ChainedLogicalStreamBuilder> configurator)
                                                                         throws SQLException

Throws:

SQLException

hasMinimumVersion

private Boolean hasMinimumVersion(int version)

close

public void close()

Specified by:

close in interface AutoCloseable

Overrides:

close in class JdbcConnection

close

public void close(boolean dropSlot)

reconnect

public void reconnect()
               throws SQLException

Specified by:

reconnect in interface ReplicationConnection

Throws:

SQLException