Package io.debezium.connector.postgresql.connection

Class PostgresConnection

java.lang.Object

io.debezium.jdbc.JdbcConnection

io.debezium.connector.postgresql.connection.PostgresConnection

All Implemented Interfaces:

AutoCloseable

public class PostgresConnection
extends JdbcConnection

JdbcConnection connection extension used for connecting to Postgres instances.

Author:

Horia Chiorean

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static interface	PostgresConnection.PostgresValueConverterBuilder

Nested classes/interfaces inherited from class io.debezium.jdbc.JdbcConnection

JdbcConnection.BlockingMultiResultSetConsumer, JdbcConnection.BlockingResultSetConsumer, JdbcConnection.CallPreparer, JdbcConnection.ConnectionFactory, JdbcConnection.MultiResultSetConsumer, JdbcConnection.Operations, JdbcConnection.ParameterResultSetConsumer, JdbcConnection.ResultSetConsumer, JdbcConnection.ResultSetExtractor<T extends Object>, JdbcConnection.ResultSetMapper<T extends Object>, JdbcConnection.StatementFactory, JdbcConnection.StatementPreparer

Field Summary

Fields

Modifier and Type	Field	Description
static final String	CONNECTION_DROP_SLOT
static final String	CONNECTION_GENERAL
static final String	CONNECTION_HEARTBEAT
static final String	CONNECTION_SLOT_INFO
static final String	CONNECTION_STREAMING
static final String	CONNECTION_VALIDATE_CONNECTION
private final PostgresDefaultValueConverter	defaultValueConverter
private static final Pattern	EXPRESSION_DEFAULT_PATTERN
protected static final JdbcConnection.ConnectionFactory	FACTORY
private static final Pattern	FUNCTION_DEFAULT_PATTERN
private static org.slf4j.Logger	LOGGER
private static final int	MAX_ATTEMPTS_FOR_OBTAINING_REPLICATION_SLOT	Obtaining a replication slot may fail if there's a pending transaction.
private static final Duration	PAUSE_BETWEEN_REPLICATION_SLOT_RETRIEVAL_ATTEMPTS
private final TypeRegistry	typeRegistry
private static final String	URL_PATTERN

Constructor Summary

Constructors

Constructor	Description
PostgresConnection(PostgresConnectorConfig config, TypeRegistry typeRegistry, String connectionUsage)	Create a Postgres connection using the supplied configuration and TypeRegistry
PostgresConnection(JdbcConfiguration config, PostgresConnection.PostgresValueConverterBuilder valueConverterBuilder, String connectionUsage)	Creates a Postgres connection using the supplied configuration.
PostgresConnection(JdbcConfiguration config, String connectionUsage)	Creates a Postgres connection using the supplied configuration.

Method Summary

Modifier and Type	Method	Description
(package private) static JdbcConfiguration	addDefaultSettings(JdbcConfiguration configuration, String connectionUsage)
void	close()
String	connectionString()	Returns a JDBC connection string for the current configuration.
Long	currentTransactionId()	Returns the PG id of the current active transaction
long	currentXLogLocation()	Returns the current position in the server tx log.
private Optional<ColumnEditor>	doReadTableColumn(ResultSet columnMetadata, TableId tableId, Tables.ColumnNameFilter columnFilter)
boolean	dropPublication(String publicationName)	Drops the debezium publication that was created.
boolean	dropReplicationSlot(String slotName)	Drops a replication slot that was created on the DB
private ServerInfo.ReplicationSlot	fetchReplicationSlotInfo(String slotName, String pluginName)	Fetches the state of a replication stage given a slot name and plugin name
Set<TableId>	getAllTableIds(String catalogName)	Retrieves all TableIds in a given database catalog, including partitioned tables.
Object	getColumnValue(ResultSet rs, int columnIndex, Column column, Table table)
Charset	getDatabaseCharset()
PostgresDefaultValueConverter	getDefaultValueConverter()
SlotState	getReplicationSlotState(String slotName, String pluginName)	Returns the current state of the replication slot
org.postgresql.jdbc.TimestampUtils	getTimestampUtils()
TypeRegistry	getTypeRegistry()
private boolean	hasIdleTransactions()
protected boolean	isTableType(String tableType)
protected boolean	isTableUniqueIndexIncluded(String indexName, String columnName)
private Lsn	parseConfirmedFlushLsn(String slotName, String pluginName, String database, ResultSet rs)	Obtains the LSN to resume streaming from.
private Lsn	parseRestartLsn(String slotName, String pluginName, String database, ResultSet rs)
protected ServerInfo.ReplicationSlot	queryForSlot(String slotName, String database, String pluginName, JdbcConnection.ResultSetMapper<ServerInfo.ReplicationSlot> map)
String	quotedColumnIdString(String columnName)
Optional<Column>	readColumnForDecoder(ResultSet columnMetadata, TableId tableId, Tables.ColumnNameFilter columnNameFilter)
String	readIndexOfReplicaIdentity(TableId tableId)	This query retrieves information about the INDEX as long as replica identity is configure USING INDEX
ReplicaIdentityInfo	readReplicaIdentityInfo(TableId tableId)	Prints out information about the REPLICA IDENTITY status of a table.
(package private) ServerInfo.ReplicationSlot	readReplicationSlotInfo(String slotName, String pluginName)	Fetches a replication slot, repeating the query until either the slot is created or until the max number of attempts has been reached To fetch the slot without the retries, use the fetchReplicationSlotInfo(java.lang.String, java.lang.String) call
protected Optional<ColumnEditor>	readTableColumn(ResultSet columnMetadata, TableId tableId, Tables.ColumnNameFilter columnFilter)
protected int	resolveJdbcType(int metadataJdbcType, int nativeType)
protected int	resolveNativeType(String typeName)
ServerInfo	serverInfo()	Returns information about the PG server to which this instance is connected.
void	setQueryColumnValue(PreparedStatement statement, Column column, int pos, Object value)
void	setReplicaIdentityForTable(TableId tableId, ReplicaIdentityInfo replicaIdentityValue)	Update REPLICA IDENTITY status of a table.
protected String[]	supportedTableTypes()
private Lsn	tryFallbackToRestartLsn(String slotName, String pluginName, String database, ResultSet rs)
private Lsn	tryParseLsn(String slotName, String pluginName, String database, ResultSet rs, String column)
private static void	validateServerVersion(Statement statement)

Methods inherited from class io.debezium.jdbc.JdbcConnection

buildSelectWithRowLimits, call, commit, config, connect, connection, connection, connectionString, database, escapeEscapeSequence, execute, execute, executeWithoutCommitting, getAttributeDetails, getColumnsDetails, getCurrentTimestamp, isConnected, isNullable, isValid, loadKeyStore, overrideColumn, parseSqlStatementString, patternBasedFactory, patternBasedFactory, prepareQuery, prepareQuery, prepareQuery, prepareQuery, prepareQuery, prepareQueryAndMap, prepareQueryWithBlockingConsumer, prepareUpdate, print, print, query, query, queryAndMap, queryAndMap, querySingleValue, queryWithBlockingConsumer, quotedTableIdString, readAllCatalogNames, readAllSchemaNames, readAllTableNames, readPrimaryKeyNames, readPrimaryKeyOrUniqueIndexNames, readSchema, readTableNames, readTablePreparedStatement, readTableStatement, readTableUniqueIndices, resolveCatalogName, rollback, rowToArray, setAutoCommit, singleResultMapper, tableTypes, username

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

CONNECTION_STREAMING

public static final String CONNECTION_STREAMING

See Also:

• Constant Field Values

CONNECTION_SLOT_INFO

public static final String CONNECTION_SLOT_INFO

See Also:

• Constant Field Values

CONNECTION_DROP_SLOT

public static final String CONNECTION_DROP_SLOT

See Also:

• Constant Field Values

CONNECTION_VALIDATE_CONNECTION

public static final String CONNECTION_VALIDATE_CONNECTION

See Also:

• Constant Field Values

CONNECTION_HEARTBEAT

public static final String CONNECTION_HEARTBEAT

See Also:

• Constant Field Values

CONNECTION_GENERAL

public static final String CONNECTION_GENERAL

See Also:

• Constant Field Values

FUNCTION_DEFAULT_PATTERN

private static final Pattern FUNCTION_DEFAULT_PATTERN

EXPRESSION_DEFAULT_PATTERN

private static final Pattern EXPRESSION_DEFAULT_PATTERN

LOGGER

private static org.slf4j.Logger LOGGER

URL_PATTERN

private static final String URL_PATTERN

FACTORY

protected static final JdbcConnection.ConnectionFactory FACTORY

MAX_ATTEMPTS_FOR_OBTAINING_REPLICATION_SLOT

private static final int MAX_ATTEMPTS_FOR_OBTAINING_REPLICATION_SLOT

Obtaining a replication slot may fail if there's a pending transaction. We're retrying to get a slot for 30 min.

See Also:

• Constant Field Values

PAUSE_BETWEEN_REPLICATION_SLOT_RETRIEVAL_ATTEMPTS

private static final Duration PAUSE_BETWEEN_REPLICATION_SLOT_RETRIEVAL_ATTEMPTS

typeRegistry

private final TypeRegistry typeRegistry

defaultValueConverter

private final PostgresDefaultValueConverter defaultValueConverter

Constructor Details

PostgresConnection

public PostgresConnection(JdbcConfiguration config,
 PostgresConnection.PostgresValueConverterBuilder valueConverterBuilder,
 String connectionUsage)

Creates a Postgres connection using the supplied configuration. If necessary this connection is able to resolve data type mappings. Such a connection requires a PostgresValueConverter, and will provide its own TypeRegistry. Usually only one such connection per connector is needed.

Parameters:

config - Configuration instance, may not be null.

valueConverterBuilder - supplies a configured PostgresValueConverter for a given TypeRegistry

connectionUsage - a symbolic name of the connection to be tracked in monitoring tools

PostgresConnection

public PostgresConnection(PostgresConnectorConfig config,
 TypeRegistry typeRegistry,
 String connectionUsage)

Create a Postgres connection using the supplied configuration and TypeRegistry

Parameters:

config - Configuration instance, may not be null.

typeRegistry - an existing/already-primed TypeRegistry instance

connectionUsage - a symbolic name of the connection to be tracked in monitoring tools

PostgresConnection

public PostgresConnection(JdbcConfiguration config,
 String connectionUsage)

Creates a Postgres connection using the supplied configuration. The connector is the regular one without datatype resolution capabilities.

Parameters:

config - Configuration instance, may not be null.

connectionUsage - a symbolic name of the connection to be tracked in monitoring tools

Method Details

addDefaultSettings

static JdbcConfiguration addDefaultSettings(JdbcConfiguration configuration,
 String connectionUsage)

connectionString

public String connectionString()

Returns a JDBC connection string for the current configuration.

Returns:

a String where the variables in urlPattern are replaced with values from the configuration

readReplicaIdentityInfo

@VisibleForTesting
public ReplicaIdentityInfo readReplicaIdentityInfo(TableId tableId)
                                            throws SQLException

Prints out information about the REPLICA IDENTITY status of a table. This in turn determines how much information is available for UPDATE and DELETE operations for logical replication.

Parameters:

tableId - the identifier of the table

Returns:

the replica identity information; never null

Throws:

SQLException - if there is a problem obtaining the replica identity information for the given table

readIndexOfReplicaIdentity

@VisibleForTesting
public String readIndexOfReplicaIdentity(TableId tableId)
                                  throws SQLException

This query retrieves information about the INDEX as long as replica identity is configure USING INDEX

Parameters:

tableId - the identifier of the table

Returns:

Index name linked to replica identity; never null

Throws:

SQLException - if there is a problem obtaining the replica identity and index information for the given table

setReplicaIdentityForTable

public void setReplicaIdentityForTable(TableId tableId,
 ReplicaIdentityInfo replicaIdentityValue)

Update REPLICA IDENTITY status of a table. This in turn determines how much information is available for UPDATE and DELETE operations for logical replication.

Parameters:

tableId - the identifier of the table

replicaIdentityValue - Replica Identity value

getReplicationSlotState

public SlotState getReplicationSlotState(String slotName,
 String pluginName)
                                  throws SQLException

Returns the current state of the replication slot

Parameters:

slotName - the name of the slot

pluginName - the name of the plugin used for the desired slot

Returns:

the SlotState or null, if no slot state is found

Throws:

SQLException

fetchReplicationSlotInfo

private ServerInfo.ReplicationSlot fetchReplicationSlotInfo(String slotName,
 String pluginName)
                                                     throws SQLException

Fetches the state of a replication stage given a slot name and plugin name

Parameters:

slotName - the name of the slot

pluginName - the name of the plugin used for the desired slot

Returns:

the ServerInfo.ReplicationSlot object or a ServerInfo.ReplicationSlot.INVALID if the slot is not valid

Throws:

SQLException - is thrown by the underlying JDBC

readReplicationSlotInfo

@VisibleForTesting
ServerInfo.ReplicationSlot readReplicationSlotInfo(String slotName,
 String pluginName)
                                            throws SQLException,
InterruptedException

Fetches a replication slot, repeating the query until either the slot is created or until the max number of attempts has been reached To fetch the slot without the retries, use the fetchReplicationSlotInfo(java.lang.String, java.lang.String) call

Parameters:

slotName - the slot name

pluginName - the name of the plugin

Returns:

the ServerInfo.ReplicationSlot object or a ServerInfo.ReplicationSlot.INVALID if the slot is not valid

Throws:

SQLException - is thrown by the underyling jdbc driver

InterruptedException - is thrown if we don't return an answer within the set number of retries

queryForSlot

protected ServerInfo.ReplicationSlot queryForSlot(String slotName,
 String database,
 String pluginName,
 JdbcConnection.ResultSetMapper<ServerInfo.ReplicationSlot> map)
                                           throws SQLException

Throws:

SQLException

parseConfirmedFlushLsn

private Lsn parseConfirmedFlushLsn(String slotName,
 String pluginName,
 String database,
 ResultSet rs)

Obtains the LSN to resume streaming from. On PG 9.5 there is no confirmed_flushed_lsn yet, so restart_lsn will be read instead. This may result in more records to be re-read after a restart.

hasIdleTransactions

private boolean hasIdleTransactions()
                             throws SQLException

Throws:

SQLException

tryFallbackToRestartLsn

private Lsn tryFallbackToRestartLsn(String slotName,
 String pluginName,
 String database,
 ResultSet rs)

parseRestartLsn

private Lsn parseRestartLsn(String slotName,
 String pluginName,
 String database,
 ResultSet rs)

tryParseLsn

private Lsn tryParseLsn(String slotName,
 String pluginName,
 String database,
 ResultSet rs,
 String column)
                 throws org.apache.kafka.connect.errors.ConnectException,
SQLException

Throws:

org.apache.kafka.connect.errors.ConnectException

SQLException

dropReplicationSlot

public boolean dropReplicationSlot(String slotName)

Drops a replication slot that was created on the DB

Parameters:

slotName - the name of the replication slot, may not be null

Returns:

true if the slot was dropped, false otherwise

dropPublication

public boolean dropPublication(String publicationName)

Drops the debezium publication that was created.

Parameters:

publicationName - the publication name, may not be null

Returns:

true if the publication was dropped, false otherwise

close

public void close()

Specified by:

close in interface AutoCloseable

Overrides:

close in class JdbcConnection

currentTransactionId

public Long currentTransactionId()
                          throws SQLException

Returns the PG id of the current active transaction

Returns:

a PG transaction identifier, or null if no tx is active

Throws:

SQLException - if anything fails.

currentXLogLocation

public long currentXLogLocation()
                         throws SQLException

Returns the current position in the server tx log.

Returns:

a long value, never negative

Throws:

SQLException - if anything unexpected fails.

serverInfo

public ServerInfo serverInfo()
                      throws SQLException

Returns information about the PG server to which this instance is connected.

Returns:

a ServerInfo instance, never null

Throws:

SQLException - if anything fails

getDatabaseCharset

public Charset getDatabaseCharset()

getTimestampUtils

public org.postgresql.jdbc.TimestampUtils getTimestampUtils()

validateServerVersion

private static void validateServerVersion(Statement statement)
                                   throws SQLException

Throws:

SQLException

quotedColumnIdString

public String quotedColumnIdString(String columnName)

Overrides:

quotedColumnIdString in class JdbcConnection

resolveNativeType

protected int resolveNativeType(String typeName)

Overrides:

resolveNativeType in class JdbcConnection

resolveJdbcType

protected int resolveJdbcType(int metadataJdbcType,
 int nativeType)

Overrides:

resolveJdbcType in class JdbcConnection

readTableColumn

protected Optional<ColumnEditor> readTableColumn(ResultSet columnMetadata,
 TableId tableId,
 Tables.ColumnNameFilter columnFilter)
                                          throws SQLException

Overrides:

readTableColumn in class JdbcConnection

Throws:

SQLException

readColumnForDecoder

public Optional<Column> readColumnForDecoder(ResultSet columnMetadata,
 TableId tableId,
 Tables.ColumnNameFilter columnNameFilter)
                                      throws SQLException

Throws:

SQLException

doReadTableColumn

private Optional<ColumnEditor> doReadTableColumn(ResultSet columnMetadata,
 TableId tableId,
 Tables.ColumnNameFilter columnFilter)
                                          throws SQLException

Throws:

SQLException

getDefaultValueConverter

public PostgresDefaultValueConverter getDefaultValueConverter()

getTypeRegistry

public TypeRegistry getTypeRegistry()

getColumnValue

public Object getColumnValue(ResultSet rs,
 int columnIndex,
 Column column,
 Table table)
                      throws SQLException

Overrides:

getColumnValue in class JdbcConnection

Throws:

SQLException

supportedTableTypes

protected String[] supportedTableTypes()

Overrides:

supportedTableTypes in class JdbcConnection

isTableType

protected boolean isTableType(String tableType)

Overrides:

isTableType in class JdbcConnection

isTableUniqueIndexIncluded

protected boolean isTableUniqueIndexIncluded(String indexName,
 String columnName)

Overrides:

isTableUniqueIndexIncluded in class JdbcConnection

getAllTableIds

public Set<TableId> getAllTableIds(String catalogName)
                            throws SQLException

Retrieves all TableIds in a given database catalog, including partitioned tables.

Parameters:

catalogName - the catalog/database name

Returns:

set of all table ids for existing table objects

Throws:

SQLException - if a database exception occurred

setQueryColumnValue

public void setQueryColumnValue(PreparedStatement statement,
 Column column,
 int pos,
 Object value)
                         throws SQLException

Overrides:

setQueryColumnValue in class JdbcConnection

Throws:

SQLException