io.debezium.connector.postgresql.connection.pgoutput

Class PgOutputMessageDecoder

java.lang.Object

io.debezium.connector.postgresql.connection.AbstractMessageDecoder

io.debezium.connector.postgresql.connection.pgoutput.PgOutputMessageDecoder

All Implemented Interfaces:

MessageDecoder

public class PgOutputMessageDecoder
extends AbstractMessageDecoder

Decodes messages from the PG logical replication plug-in ("pgoutput"). See https://www.postgresql.org/docs/10/protocol-logicalrep-message-formats.html for the protocol specification.

Author:

Gunnar Morling, Chris Cranford

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	PgOutputMessageDecoder.MessageType

Field Summary

Fields

Modifier and Type	Field and Description
private Instant	commitTimestamp
private MessageDecoderConfig	config
private static org.slf4j.Logger	LOGGER
private static Instant	PG_EPOCH
private static byte	SPACE
private int	transactionId

Constructor Summary

Constructors

Constructor and Description
PgOutputMessageDecoder(MessageDecoderConfig config)

Method Summary

Modifier and Type	Method and Description
private void	decodeDelete(ByteBuffer buffer, TypeRegistry typeRegistry, ReplicationStream.ReplicationMessageProcessor processor) Callback handler for the 'D' delete replication stream message.
private void	decodeInsert(ByteBuffer buffer, TypeRegistry typeRegistry, ReplicationStream.ReplicationMessageProcessor processor) Callback handler for the 'I' insert replication stream message.
private void	decodeUpdate(ByteBuffer buffer, TypeRegistry typeRegistry, ReplicationStream.ReplicationMessageProcessor processor) Callback handler for the 'U' update replication stream message.
private Map<String,Boolean>	getTableColumnOptionalityFromDatabase(DatabaseMetaData databaseMetadata, String schemaName, String tableName)
private Set<String>	getTablePrimaryKeyColumnNamesFromDatabase(DatabaseMetaData databaseMetadata, String schemaName, String tableName)
private void	handleBeginMessage(ByteBuffer buffer) Callback handler for the 'B' begin replication message.
private void	handleCommitMessage(ByteBuffer buffer) Callback handler for the 'C' commit replication message.
private void	handleRelationMessage(ByteBuffer buffer, TypeRegistry typeRegistry) Callback handler for the 'R' relation replication message.
private boolean	isColumnInPrimaryKey(String schemaName, String tableName, String columnName, Set<String> primaryKeyColumns)
org.postgresql.replication.fluent.logical.ChainedLogicalStreamBuilder	optionsWithMetadata(org.postgresql.replication.fluent.logical.ChainedLogicalStreamBuilder builder) Allows MessageDecoder to configure options with which the replication stream is started.
org.postgresql.replication.fluent.logical.ChainedLogicalStreamBuilder	optionsWithoutMetadata(org.postgresql.replication.fluent.logical.ChainedLogicalStreamBuilder builder) Allows MessageDecoder to configure options with which the replication stream is started.
void	processMessage(ByteBuffer buffer, ReplicationStream.ReplicationMessageProcessor processor, TypeRegistry typeRegistry) Process a message upon arrival from logical decoder
private static String	readColumnValueAsString(ByteBuffer buffer) Reads the replication stream where the column stream specifies a length followed by the value.
private static String	readString(ByteBuffer buffer) Reads the replication stream up to the next null-terminator byte and returns the contents as a string.
private static List<ReplicationMessage.Column>	resolveColumnsFromStreamTupleData(ByteBuffer buffer, TypeRegistry typeRegistry, Table table) Resolve the replication stream's tuple data to a list of replication message columns.
private Optional<Table>	resolveRelation(int relationId) Resolves a given replication message relation identifier to a Table.
private Table	resolveRelationFromMetadata(PgOutputRelationMetaData metadata) Constructs a Table based on the supplied PgOutputRelationMetaData.
boolean	shouldMessageBeSkipped(ByteBuffer buffer, Long lastReceivedLsn, Long startLsn, boolean skipFirstFlushRecord) A callback into the decoder allowing it to decide whether the supplied message should be processed by the decoder or whether it can be skipped.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.debezium.connector.postgresql.connection.MessageDecoder

setContainsMetadata

Field Detail

LOGGER

private static final org.slf4j.Logger LOGGER

PG_EPOCH

private static final Instant PG_EPOCH

SPACE

private static final byte SPACE

See Also:

Constant Field Values

commitTimestamp

private Instant commitTimestamp

transactionId

private int transactionId

config

private final MessageDecoderConfig config

Constructor Detail

PgOutputMessageDecoder

public PgOutputMessageDecoder(MessageDecoderConfig config)

Method Detail

shouldMessageBeSkipped

public boolean shouldMessageBeSkipped(ByteBuffer buffer,
                                      Long lastReceivedLsn,
                                      Long startLsn,
                                      boolean skipFirstFlushRecord)

Description copied from interface: MessageDecoder

A callback into the decoder allowing it to decide whether the supplied message should be processed by the decoder or whether it can be skipped.

Specified by:

shouldMessageBeSkipped in interface MessageDecoder

Overrides:

shouldMessageBeSkipped in class AbstractMessageDecoder

Parameters:

buffer - the replication stream buffer

lastReceivedLsn - the last LSN reported by the replication stream

startLsn - the starting LSN reported by the streaming producer

skipFirstFlushRecord - whether first flush record should be skipped

Returns:

true if the incoming message should be skipped, false otherwise

processMessage

public void processMessage(ByteBuffer buffer,
                           ReplicationStream.ReplicationMessageProcessor processor,
                           TypeRegistry typeRegistry)
                    throws SQLException,
                           InterruptedException

Description copied from interface: MessageDecoder

Process a message upon arrival from logical decoder

Parameters:

buffer - - binary representation of replication message

processor - - message processing on arrival

typeRegistry - - registry with known types

Throws:

SQLException

InterruptedException

optionsWithMetadata

public org.postgresql.replication.fluent.logical.ChainedLogicalStreamBuilder optionsWithMetadata(org.postgresql.replication.fluent.logical.ChainedLogicalStreamBuilder builder)

Description copied from interface: MessageDecoder

Allows MessageDecoder to configure options with which the replication stream is started. The messages CAN contain type metadata. See PostgreSQL command START_REPLICATION SLOT for more details.

Returns:

the builder instance

optionsWithoutMetadata

public org.postgresql.replication.fluent.logical.ChainedLogicalStreamBuilder optionsWithoutMetadata(org.postgresql.replication.fluent.logical.ChainedLogicalStreamBuilder builder)

Description copied from interface: MessageDecoder

Allows MessageDecoder to configure options with which the replication stream is started. The messages MUST NOT contain type metadata. See PostgreSQL command START_REPLICATION SLOT for more details.

Returns:

the builder instance

handleBeginMessage

private void handleBeginMessage(ByteBuffer buffer)

Callback handler for the 'B' begin replication message.

Parameters:

buffer - The replication stream buffer

handleCommitMessage

private void handleCommitMessage(ByteBuffer buffer)

Callback handler for the 'C' commit replication message.

Parameters:

buffer - The replication stream buffer

handleRelationMessage

private void handleRelationMessage(ByteBuffer buffer,
                                   TypeRegistry typeRegistry)
                            throws SQLException

Callback handler for the 'R' relation replication message.

Parameters:

buffer - The replication stream buffer

typeRegistry - The postgres type registry

Throws:

SQLException

getTableColumnOptionalityFromDatabase

private Map<String,Boolean> getTableColumnOptionalityFromDatabase(DatabaseMetaData databaseMetadata,
                                                                  String schemaName,
                                                                  String tableName)

getTablePrimaryKeyColumnNamesFromDatabase

private Set<String> getTablePrimaryKeyColumnNamesFromDatabase(DatabaseMetaData databaseMetadata,
                                                              String schemaName,
                                                              String tableName)

isColumnInPrimaryKey

private boolean isColumnInPrimaryKey(String schemaName,
                                     String tableName,
                                     String columnName,
                                     Set<String> primaryKeyColumns)

decodeInsert

private void decodeInsert(ByteBuffer buffer,
                          TypeRegistry typeRegistry,
                          ReplicationStream.ReplicationMessageProcessor processor)
                   throws SQLException,
                          InterruptedException

Callback handler for the 'I' insert replication stream message.

Parameters:

buffer - The replication stream buffer

typeRegistry - The postgres type registry

processor - The replication message processor

Throws:

SQLException

InterruptedException

decodeUpdate

private void decodeUpdate(ByteBuffer buffer,
                          TypeRegistry typeRegistry,
                          ReplicationStream.ReplicationMessageProcessor processor)
                   throws SQLException,
                          InterruptedException

Callback handler for the 'U' update replication stream message.

Parameters:

buffer - The replication stream buffer

typeRegistry - The postgres type registry

processor - The replication message processor

Throws:

SQLException

InterruptedException

decodeDelete

private void decodeDelete(ByteBuffer buffer,
                          TypeRegistry typeRegistry,
                          ReplicationStream.ReplicationMessageProcessor processor)
                   throws SQLException,
                          InterruptedException

Callback handler for the 'D' delete replication stream message.

Parameters:

buffer - The replication stream buffer

typeRegistry - The postgres type registry

processor - The replication message processor

Throws:

SQLException

InterruptedException

resolveRelation

private Optional<Table> resolveRelation(int relationId)

Resolves a given replication message relation identifier to a Table.

Parameters:

relationId - The replication message stream's relation identifier

Returns:

table resolved from a prior relation message or direct lookup from the schema

resolveRelationFromMetadata

private Table resolveRelationFromMetadata(PgOutputRelationMetaData metadata)

Constructs a Table based on the supplied PgOutputRelationMetaData.

Parameters:

metadata - The relation metadata collected from previous 'R' replication stream messages

Returns:

table based on a prior replication relation message

readString

private static String readString(ByteBuffer buffer)

Reads the replication stream up to the next null-terminator byte and returns the contents as a string.

Parameters:

buffer - The replication stream buffer

Returns:

string read from the replication stream

readColumnValueAsString

private static String readColumnValueAsString(ByteBuffer buffer)

Reads the replication stream where the column stream specifies a length followed by the value.

Parameters:

buffer - The replication stream buffer

Returns:

the column value as a string read from the replication stream

resolveColumnsFromStreamTupleData

private static List<ReplicationMessage.Column> resolveColumnsFromStreamTupleData(ByteBuffer buffer,
                                                                                 TypeRegistry typeRegistry,
                                                                                 Table table)

Resolve the replication stream's tuple data to a list of replication message columns.

Parameters:

buffer - The replication stream buffer

typeRegistry - The database type registry

table - The database table

Returns:

list of replication message columns