io.debezium.connector.postgresql.connection.pgoutput

Class PgOutputMessageDecoder

java.lang.Object

io.debezium.connector.postgresql.connection.AbstractMessageDecoder

io.debezium.connector.postgresql.connection.pgoutput.PgOutputMessageDecoder

All Implemented Interfaces:

MessageDecoder

public class PgOutputMessageDecoder
extends AbstractMessageDecoder

Decodes messages from the PG logical replication plug-in ("pgoutput"). See https://www.postgresql.org/docs/10/protocol-logicalrep-message-formats.html for the protocol specification.

Author:

Gunnar Morling, Chris Cranford

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	PgOutputMessageDecoder.MessageType

Field Summary

Fields

Modifier and Type	Field and Description
private Instant	commitTimestamp
private MessageDecoderConfig	config
private static org.slf4j.Logger	LOGGER
private static Instant	PG_EPOCH
private static byte	SPACE
private int	transactionId

Constructor Summary

Constructors

Constructor and Description
PgOutputMessageDecoder(MessageDecoderConfig config)

Method Summary

Modifier and Type	Method and Description
private void	decodeDelete(ByteBuffer buffer, TypeRegistry typeRegistry, ReplicationStream.ReplicationMessageProcessor processor) Callback handler for the 'D' delete replication stream message.
private void	decodeInsert(ByteBuffer buffer, TypeRegistry typeRegistry, ReplicationStream.ReplicationMessageProcessor processor) Callback handler for the 'I' insert replication stream message.
private void	decodeTruncate(ByteBuffer buffer, TypeRegistry typeRegistry, ReplicationStream.ReplicationMessageProcessor processor) Callback handler for the 'T' truncate replication stream message.
private void	decodeUpdate(ByteBuffer buffer, TypeRegistry typeRegistry, ReplicationStream.ReplicationMessageProcessor processor) Callback handler for the 'U' update replication stream message.
private Map<String,Boolean>	getTableColumnOptionalityFromDatabase(DatabaseMetaData databaseMetadata, String schemaName, String tableName)
private List<String>	getTruncateOptions(int flag) Convert truncate option bits to postgres syntax truncate options
private void	handleBeginMessage(ByteBuffer buffer, ReplicationStream.ReplicationMessageProcessor processor) Callback handler for the 'B' begin replication message.
private void	handleCommitMessage(ByteBuffer buffer, ReplicationStream.ReplicationMessageProcessor processor) Callback handler for the 'C' commit replication message.
private void	handleRelationMessage(ByteBuffer buffer, TypeRegistry typeRegistry) Callback handler for the 'R' relation replication message.
private boolean	isColumnInPrimaryKey(String schemaName, String tableName, String columnName, List<String> primaryKeyColumns)
org.postgresql.replication.fluent.logical.ChainedLogicalStreamBuilder	optionsWithMetadata(org.postgresql.replication.fluent.logical.ChainedLogicalStreamBuilder builder) Allows MessageDecoder to configure options with which the replication stream is started.
org.postgresql.replication.fluent.logical.ChainedLogicalStreamBuilder	optionsWithoutMetadata(org.postgresql.replication.fluent.logical.ChainedLogicalStreamBuilder builder) Allows MessageDecoder to configure options with which the replication stream is started.
void	processNotEmptyMessage(ByteBuffer buffer, ReplicationStream.ReplicationMessageProcessor processor, TypeRegistry typeRegistry)
private static String	readColumnValueAsString(ByteBuffer buffer) Reads the replication stream where the column stream specifies a length followed by the value.
private static String	readString(ByteBuffer buffer) Reads the replication stream up to the next null-terminator byte and returns the contents as a string.
private static List<ReplicationMessage.Column>	resolveColumnsFromStreamTupleData(ByteBuffer buffer, TypeRegistry typeRegistry, Table table) Resolve the replication stream's tuple data to a list of replication message columns.
private Optional<Table>	resolveRelation(int relationId) Resolves a given replication message relation identifier to a Table.
private Table	resolveRelationFromMetadata(PgOutputRelationMetaData metadata) Constructs a Table based on the supplied PgOutputRelationMetaData.
boolean	shouldMessageBeSkipped(ByteBuffer buffer, Lsn lastReceivedLsn, Lsn startLsn, WalPositionLocator walPosition) A callback into the decoder allowing it to decide whether the supplied message should be processed by the decoder or whether it can be skipped.

Methods inherited from class io.debezium.connector.postgresql.connection.AbstractMessageDecoder

processMessage

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.debezium.connector.postgresql.connection.MessageDecoder

setContainsMetadata

Field Detail

LOGGER

private static final org.slf4j.Logger LOGGER

PG_EPOCH

private static final Instant PG_EPOCH

SPACE

private static final byte SPACE

See Also:

Constant Field Values

commitTimestamp

private Instant commitTimestamp

transactionId

private int transactionId

config

private final MessageDecoderConfig config

Constructor Detail

PgOutputMessageDecoder

public PgOutputMessageDecoder(MessageDecoderConfig config)

Method Detail

shouldMessageBeSkipped

public boolean shouldMessageBeSkipped(ByteBuffer buffer,
                                      Lsn lastReceivedLsn,
                                      Lsn startLsn,
                                      WalPositionLocator walPosition)

Description copied from interface: MessageDecoder

A callback into the decoder allowing it to decide whether the supplied message should be processed by the decoder or whether it can be skipped.

Specified by:

shouldMessageBeSkipped in interface MessageDecoder

Overrides:

shouldMessageBeSkipped in class AbstractMessageDecoder

Parameters:

buffer - the replication stream buffer

lastReceivedLsn - the last LSN reported by the replication stream

startLsn - the starting LSN reported by the streaming producer

walPosition - wal position from which the streaming should resume

Returns:

true if the incoming message should be skipped, false otherwise

processNotEmptyMessage

public void processNotEmptyMessage(ByteBuffer buffer,
                                   ReplicationStream.ReplicationMessageProcessor processor,
                                   TypeRegistry typeRegistry)
                            throws SQLException,
                                   InterruptedException

Specified by:

processNotEmptyMessage in class AbstractMessageDecoder

Throws:

SQLException

InterruptedException

optionsWithMetadata

public org.postgresql.replication.fluent.logical.ChainedLogicalStreamBuilder optionsWithMetadata(org.postgresql.replication.fluent.logical.ChainedLogicalStreamBuilder builder)

Description copied from interface: MessageDecoder

Allows MessageDecoder to configure options with which the replication stream is started. The messages CAN contain type metadata. See PostgreSQL command START_REPLICATION SLOT for more details.

Returns:

the builder instance

optionsWithoutMetadata

public org.postgresql.replication.fluent.logical.ChainedLogicalStreamBuilder optionsWithoutMetadata(org.postgresql.replication.fluent.logical.ChainedLogicalStreamBuilder builder)

Description copied from interface: MessageDecoder

Allows MessageDecoder to configure options with which the replication stream is started. The messages MUST NOT contain type metadata. See PostgreSQL command START_REPLICATION SLOT for more details.

Returns:

the builder instance

handleBeginMessage

private void handleBeginMessage(ByteBuffer buffer,
                                ReplicationStream.ReplicationMessageProcessor processor)
                         throws SQLException,
                                InterruptedException

Callback handler for the 'B' begin replication message.

Parameters:

buffer - The replication stream buffer

processor - The replication message processor

Throws:

SQLException

InterruptedException

handleCommitMessage

private void handleCommitMessage(ByteBuffer buffer,
                                 ReplicationStream.ReplicationMessageProcessor processor)
                          throws SQLException,
                                 InterruptedException

Callback handler for the 'C' commit replication message.

Parameters:

buffer - The replication stream buffer

processor - The replication message processor

Throws:

SQLException

InterruptedException

handleRelationMessage

private void handleRelationMessage(ByteBuffer buffer,
                                   TypeRegistry typeRegistry)
                            throws SQLException

Callback handler for the 'R' relation replication message.

Parameters:

buffer - The replication stream buffer

typeRegistry - The postgres type registry

Throws:

SQLException

getTableColumnOptionalityFromDatabase

private Map<String,Boolean> getTableColumnOptionalityFromDatabase(DatabaseMetaData databaseMetadata,
                                                                  String schemaName,
                                                                  String tableName)

isColumnInPrimaryKey

private boolean isColumnInPrimaryKey(String schemaName,
                                     String tableName,
                                     String columnName,
                                     List<String> primaryKeyColumns)

decodeInsert

private void decodeInsert(ByteBuffer buffer,
                          TypeRegistry typeRegistry,
                          ReplicationStream.ReplicationMessageProcessor processor)
                   throws SQLException,
                          InterruptedException

Callback handler for the 'I' insert replication stream message.

Parameters:

buffer - The replication stream buffer

typeRegistry - The postgres type registry

processor - The replication message processor

Throws:

SQLException

InterruptedException

decodeUpdate

private void decodeUpdate(ByteBuffer buffer,
                          TypeRegistry typeRegistry,
                          ReplicationStream.ReplicationMessageProcessor processor)
                   throws SQLException,
                          InterruptedException

Callback handler for the 'U' update replication stream message.

Parameters:

buffer - The replication stream buffer

typeRegistry - The postgres type registry

processor - The replication message processor

Throws:

SQLException

InterruptedException

decodeDelete

private void decodeDelete(ByteBuffer buffer,
                          TypeRegistry typeRegistry,
                          ReplicationStream.ReplicationMessageProcessor processor)
                   throws SQLException,
                          InterruptedException

Callback handler for the 'D' delete replication stream message.

Parameters:

buffer - The replication stream buffer

typeRegistry - The postgres type registry

processor - The replication message processor

Throws:

SQLException

InterruptedException

decodeTruncate

private void decodeTruncate(ByteBuffer buffer,
                            TypeRegistry typeRegistry,
                            ReplicationStream.ReplicationMessageProcessor processor)
                     throws SQLException,
                            InterruptedException

Callback handler for the 'T' truncate replication stream message.

Parameters:

buffer - The replication stream buffer

typeRegistry - The postgres type registry

processor - The replication message processor

Throws:

SQLException

InterruptedException

getTruncateOptions

private List<String> getTruncateOptions(int flag)

Convert truncate option bits to postgres syntax truncate options

Parameters:

flag - truncate option bits

Returns:

truncate flags

resolveRelation

private Optional<Table> resolveRelation(int relationId)

Resolves a given replication message relation identifier to a Table.

Parameters:

relationId - The replication message stream's relation identifier

Returns:

table resolved from a prior relation message or direct lookup from the schema or empty when the table is filtered

resolveRelationFromMetadata

private Table resolveRelationFromMetadata(PgOutputRelationMetaData metadata)

Constructs a Table based on the supplied PgOutputRelationMetaData.

Parameters:

metadata - The relation metadata collected from previous 'R' replication stream messages

Returns:

table based on a prior replication relation message

readString

private static String readString(ByteBuffer buffer)

Reads the replication stream up to the next null-terminator byte and returns the contents as a string.

Parameters:

buffer - The replication stream buffer

Returns:

string read from the replication stream

readColumnValueAsString

private static String readColumnValueAsString(ByteBuffer buffer)

Reads the replication stream where the column stream specifies a length followed by the value.

Parameters:

buffer - The replication stream buffer

Returns:

the column value as a string read from the replication stream

resolveColumnsFromStreamTupleData

private static List<ReplicationMessage.Column> resolveColumnsFromStreamTupleData(ByteBuffer buffer,
                                                                                 TypeRegistry typeRegistry,
                                                                                 Table table)

Resolve the replication stream's tuple data to a list of replication message columns.

Parameters:

buffer - The replication stream buffer

typeRegistry - The database type registry

table - The database table

Returns:

list of replication message columns