Package io.debezium.connector.postgresql.connection

Class PostgresReplicationConnection

java.lang.Object

io.debezium.jdbc.JdbcConnection

io.debezium.connector.postgresql.connection.PostgresReplicationConnection

All Implemented Interfaces:

ReplicationConnection, AutoCloseable

public class PostgresReplicationConnection
extends JdbcConnection
implements ReplicationConnection

Implementation of a ReplicationConnection for Postgresql. Note that replication connections in PG cannot execute regular statements but only a limited number of replication-related commands.

Author:

Horia Chiorean (hchiorea@redhat.com)

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
protected static class	PostgresReplicationConnection.ReplicationConnectionBuilder

Nested classes/interfaces inherited from class io.debezium.jdbc.JdbcConnection

JdbcConnection.BlockingMultiResultSetConsumer, JdbcConnection.BlockingResultSetConsumer, JdbcConnection.CallPreparer, JdbcConnection.ConnectionFactory, JdbcConnection.MultiResultSetConsumer, JdbcConnection.Operations, JdbcConnection.ParameterResultSetConsumer, JdbcConnection.ResultSetConsumer, JdbcConnection.ResultSetExtractor<T extends Object>, JdbcConnection.ResultSetMapper<T extends Object>, JdbcConnection.StatementFactory, JdbcConnection.StatementPreparer

Nested classes/interfaces inherited from interface io.debezium.connector.postgresql.connection.ReplicationConnection

ReplicationConnection.Builder

Field Summary

Fields

Modifier and Type	Field	Description
private Lsn	defaultStartingPos
private boolean	dropSlotOnClose
private boolean	hasInitedSlot
private static org.slf4j.Logger	LOGGER
private MessageDecoder	messageDecoder
private PostgresConnectorConfig	originalConfig
private PostgresConnectorConfig.LogicalDecoder	plugin
private PostgresConnectorConfig.AutoCreateMode	publicationAutocreateMode
private String	publicationName
private SlotCreationResult	slotCreationInfo
private String	slotName
private Duration	statusUpdateInterval
private Properties	streamParams
private RelationalTableFilters	tableFilter
private TypeRegistry	typeRegistry

Constructor Summary

Constructors

Modifier	Constructor	Description
private	PostgresReplicationConnection(PostgresConnectorConfig config, String slotName, String publicationName, RelationalTableFilters tableFilter, PostgresConnectorConfig.AutoCreateMode publicationAutocreateMode, PostgresConnectorConfig.LogicalDecoder plugin, boolean dropSlotOnClose, boolean doSnapshot, Duration statusUpdateInterval, TypeRegistry typeRegistry, Properties streamParams, PostgresSchema schema)	Creates a new replication connection with the given params.

Method Summary

Modifier and Type	Method	Description
void	close()
void	close(boolean dropSlot)
Optional<SlotCreationResult>	createReplicationSlot()	Creates a new replication slot with the given option and returns the result of the command, which may contain results (depending on postgres versions)
private ReplicationStream	createReplicationStream(Lsn startLsn, WalPositionLocator walPosition)
protected static void	defaultSettings(Configuration.Builder builder)
private Set<TableId>	determineCapturedTables()
private ServerInfo.ReplicationSlot	getSlotInfo()
void	initConnection()	Forces the connection to be created, is called by startStreaming, but can be called manually in cases where we want the connection but not to to start streaming yet.
protected void	initPublication()
protected void	initReplicationSlot()
private SlotCreationResult	parseSlotCreation(ResultSet rs)
protected org.postgresql.core.BaseConnection	pgConnection()
void	reconnect()
private org.postgresql.replication.PGReplicationStream	startPgReplicationStream(Lsn lsn, Function<org.postgresql.replication.fluent.logical.ChainedLogicalStreamBuilder,org.postgresql.replication.fluent.logical.ChainedLogicalStreamBuilder> configurator)
ReplicationStream	startStreaming(Lsn offset, WalPositionLocator walPosition)	Opens a stream for reading logical replication changes from a given LSN position.
ReplicationStream	startStreaming(WalPositionLocator walPosition)	creating a replication connection and starting to stream involves a few steps: 1.
private boolean	useTemporarySlot()

Methods inherited from class io.debezium.jdbc.JdbcConnection

buildSelectWithRowLimits, call, commit, config, connect, connection, connection, connectionString, database, execute, execute, executeWithoutCommitting, getColumnValue, getDefaultValue, isConnected, isNullable, isTableUniqueIndexIncluded, isValid, parseSqlStatementString, patternBasedFactory, patternBasedFactory, prepareQuery, prepareQuery, prepareQuery, prepareQuery, prepareQuery, prepareQueryAndMap, prepareQueryWithBlockingConsumer, prepareUpdate, print, print, query, query, queryAndMap, queryAndMap, querySingleValue, queryWithBlockingConsumer, quotedColumnIdString, quotedTableIdString, readAllCatalogNames, readAllSchemaNames, readAllTableNames, readPrimaryKeyNames, readPrimaryKeyOrUniqueIndexNames, readSchema, readTableColumn, readTableNames, readTablePreparedStatement, readTableStatement, readTableUniqueIndices, resolveJdbcType, resolveNativeType, rollback, rowToArray, setAutoCommit, singleResultMapper, tableTypes, username

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.debezium.connector.postgresql.connection.ReplicationConnection

isConnected

Field Detail

LOGGER

private static org.slf4j.Logger LOGGER

slotName

private final String slotName

publicationName

private final String publicationName

tableFilter

private final RelationalTableFilters tableFilter

publicationAutocreateMode

private final PostgresConnectorConfig.AutoCreateMode publicationAutocreateMode

plugin

private final PostgresConnectorConfig.LogicalDecoder plugin

dropSlotOnClose

private final boolean dropSlotOnClose

originalConfig

private final PostgresConnectorConfig originalConfig

statusUpdateInterval

private final Duration statusUpdateInterval

messageDecoder

private final MessageDecoder messageDecoder

typeRegistry

private final TypeRegistry typeRegistry

streamParams

private final Properties streamParams

defaultStartingPos

private Lsn defaultStartingPos

slotCreationInfo

private SlotCreationResult slotCreationInfo

hasInitedSlot

private boolean hasInitedSlot

Constructor Detail

PostgresReplicationConnection

private PostgresReplicationConnection(PostgresConnectorConfig config,
                                      String slotName,
                                      String publicationName,
                                      RelationalTableFilters tableFilter,
                                      PostgresConnectorConfig.AutoCreateMode publicationAutocreateMode,
                                      PostgresConnectorConfig.LogicalDecoder plugin,
                                      boolean dropSlotOnClose,
                                      boolean doSnapshot,
                                      Duration statusUpdateInterval,
                                      TypeRegistry typeRegistry,
                                      Properties streamParams,
                                      PostgresSchema schema)

Creates a new replication connection with the given params.

Parameters:

config - the JDBC configuration for the connection; may not be null

slotName - the name of the DB slot for logical replication; may not be null

publicationName - the name of the DB publication for logical replication; may not be null

tableFilter - the tables to watch of the DB publication for logical replication; may not be null

publicationAutocreateMode - the mode for publication autocreation; may not be null

plugin - decoder matching the server side plug-in used for streaming changes; may not be null

dropSlotOnClose - whether the replication slot should be dropped once the connection is closed

statusUpdateInterval - the interval at which the replication connection should periodically send status

doSnapshot - whether the connector is doing snapshot

typeRegistry - registry with PostgreSQL types

streamParams - additional parameters to pass to the replication stream

schema - the schema; must not be null

updates to the server

Method Detail

getSlotInfo

private ServerInfo.ReplicationSlot getSlotInfo()
                                        throws SQLException,
                                               InterruptedException

Throws:

SQLException

InterruptedException

initPublication

protected void initPublication()

determineCapturedTables

private Set<TableId> determineCapturedTables()
                                      throws Exception

Throws:

Exception

initReplicationSlot

protected void initReplicationSlot()
                            throws SQLException,
                                   InterruptedException

Throws:

SQLException

InterruptedException

useTemporarySlot

private boolean useTemporarySlot()
                          throws SQLException

Throws:

SQLException

startStreaming

public ReplicationStream startStreaming(WalPositionLocator walPosition)
                                 throws SQLException,
                                        InterruptedException

creating a replication connection and starting to stream involves a few steps: 1. we create the connection and ensure that a. the slot exists b. the slot isn't currently being used 2. we query to get our potential start position in the slot (lsn) 3. we try and start streaming, depending on our options (such as in wal2json) this may fail, which can result in the connection being killed and we need to start the process over if we are using a temporary slot 4. actually start the streamer

This method takes care of all of these and this method queries for a default starting position If you know where you are starting from you should call startStreaming(Lsn, WalPositionLocator), this method delegates to that method

Specified by:

startStreaming in interface ReplicationConnection

Returns:

Throws:

SQLException

InterruptedException

startStreaming

public ReplicationStream startStreaming(Lsn offset,
                                        WalPositionLocator walPosition)
                                 throws SQLException,
                                        InterruptedException

Description copied from interface: ReplicationConnection

Opens a stream for reading logical replication changes from a given LSN position.

Note that it is possible for a server to have recycled old WAL segments (see the wal_keep_segments setting). If that is the case, then even though a LSN number may be valid, the server will not stream back any changes because they are not available.

Specified by:

startStreaming in interface ReplicationConnection

Parameters:

offset - a value representing the WAL sequence number where replication should start from; if the value is null or negative, this behaves exactly like #startStreaming().

Returns:

a PGReplicationStream from which data is read; never null

Throws:

SQLException - if anything fails

InterruptedException

See Also:

LogSequenceNumber

initConnection

public void initConnection()
                    throws SQLException,
                           InterruptedException

Description copied from interface: ReplicationConnection

Forces the connection to be created, is called by startStreaming, but can be called manually in cases where we want the connection but not to to start streaming yet. Can be called multiple times

Specified by:

initConnection in interface ReplicationConnection

Throws:

SQLException

InterruptedException

createReplicationSlot

public Optional<SlotCreationResult> createReplicationSlot()
                                                   throws SQLException

Description copied from interface: ReplicationConnection

Creates a new replication slot with the given option and returns the result of the command, which may contain results (depending on postgres versions)

Specified by:

createReplicationSlot in interface ReplicationConnection

Throws:

SQLException - if anything fails

pgConnection

protected org.postgresql.core.BaseConnection pgConnection()
                                                   throws SQLException

Throws:

SQLException

parseSlotCreation

private SlotCreationResult parseSlotCreation(ResultSet rs)

createReplicationStream

private ReplicationStream createReplicationStream(Lsn startLsn,
                                                  WalPositionLocator walPosition)
                                           throws SQLException,
                                                  InterruptedException

Throws:

SQLException

InterruptedException

startPgReplicationStream

private org.postgresql.replication.PGReplicationStream startPgReplicationStream(Lsn lsn,
                                                                                Function<org.postgresql.replication.fluent.logical.ChainedLogicalStreamBuilder,org.postgresql.replication.fluent.logical.ChainedLogicalStreamBuilder> configurator)
                                                                         throws SQLException

Throws:

SQLException

close

public void close()

Specified by:

close in interface AutoCloseable

Overrides:

close in class JdbcConnection

close

public void close(boolean dropSlot)

reconnect

public void reconnect()
               throws SQLException

Specified by:

reconnect in interface ReplicationConnection

Throws:

SQLException

defaultSettings

protected static void defaultSettings(Configuration.Builder builder)