io.debezium.connector.postgresql.connection.wal2json

Class StreamingWal2JsonMessageDecoder

java.lang.Object

io.debezium.connector.postgresql.connection.AbstractMessageDecoder

io.debezium.connector.postgresql.connection.wal2json.StreamingWal2JsonMessageDecoder

All Implemented Interfaces:

MessageDecoder

public class StreamingWal2JsonMessageDecoder
extends AbstractMessageDecoder

JSON deserialization of a message sent by wal2json logical decoding plugin. The plugin sends all changes in one transaction as a single batch and they are passed to processor one-by-one. The JSON file arrives in chunks of a big JSON file where the chunks are not valid JSON itself.

There are four different chunks that can arrive from the decoder. Beginning of message

 {
   "xid": 563,
   "timestamp": "2018-03-20 10:58:43.396355+01",
   "change": [
 

First change

      {
          "kind": "insert",
          "schema": "public",
          "table": "numeric_decimal_table",
          "columnnames": ["pk", "d", "dzs", "dvs", "d_nn", "n", "nzs", "nvs", "d_int", "dzs_int", "dvs_int", "n_int", "nzs_int", "nvs_int", "d_nan", "dzs_nan", "dvs_nan", "n_nan", "nzs_nan", "nvs_nan"],
          "columntypes": ["integer", "numeric(3,2)", "numeric(4,0)", "numeric", "numeric(3,2)", "numeric(6,4)", "numeric(4,0)", "numeric", "numeric(3,2)", "numeric(4,0)", "numeric", "numeric(6,4)", "numeric(4,0)", "numeric", "numeric(3,2)", "numeric(4,0)", "numeric", "numeric(6,4)", "numeric(4,0)", "numeric"],
          "columnoptionals": [false, true, true, true, false, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true],
          "columnvalues": [1, 1.10, 10, 10.1111, 3.30, 22.2200, 22, 22.2222, 1.00, 10, 10, 22.0000, 22, 22, null, null, null, null, null, null]
      }
 

Further changes

      ,{
          "kind": "insert",
          "schema": "public",
          "table": "numeric_decimal_table",
          "columnnames": ["pk", "d", "dzs", "dvs", "d_nn", "n", "nzs", "nvs", "d_int", "dzs_int", "dvs_int", "n_int", "nzs_int", "nvs_int", "d_nan", "dzs_nan", "dvs_nan", "n_nan", "nzs_nan", "nvs_nan"],
          "columntypes": ["integer", "numeric(3,2)", "numeric(4,0)", "numeric", "numeric(3,2)", "numeric(6,4)", "numeric(4,0)", "numeric", "numeric(3,2)", "numeric(4,0)", "numeric", "numeric(6,4)", "numeric(4,0)", "numeric", "numeric(3,2)", "numeric(4,0)", "numeric", "numeric(6,4)", "numeric(4,0)", "numeric"],
          "columnoptionals": [false, true, true, true, false, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true],
          "columnvalues": [1, 1.10, 10, 10.1111, 3.30, 22.2200, 22, 22.2222, 1.00, 10, 10, 22.0000, 22, 22, null, null, null, null, null, null]
      }
 

End of message

      ]
 }
 

For parsing purposes it is necessary to add or remove a fragment of JSON to make a well-formatted JSON out of it. The last message is just dropped.

Author:

Jiri Pechanec

Field Summary

Fields

Modifier and Type	Field and Description
private static byte	COMMA
private Instant	commitTime
private boolean	containsMetadata
private static byte	CR
private byte[]	currentChunk To identify if the last current chunk is the last one we can send the current one for processing only after we read the next one or the end of message fragment.
private DateTimeFormat	dateTime
private static byte	LEFT_BRACE
private static org.slf4j.Logger	LOGGER
private boolean	messageInProgress
private static byte	RIGHT_BRACE
private static byte	RIGHT_BRACKET
private static byte	SPACE
private static byte	TAB
private long	txId
private static long	UNDEFINED_LONG

Constructor Summary

Constructors

Constructor and Description
StreamingWal2JsonMessageDecoder(MessageDecoderConfig config)

Method Summary

Modifier and Type	Method and Description
private void	doProcessMessage(ReplicationStream.ReplicationMessageProcessor processor, TypeRegistry typeRegistry, byte[] content, boolean lastMessage)
private byte	getFirstNonWhiteChar(byte[] array)
private byte	getLastNonWhiteChar(byte[] array)
private boolean	isWhitespace(byte c)
protected void	nonInitialChunk(ReplicationStream.ReplicationMessageProcessor processor, TypeRegistry typeRegistry, byte[] content)
org.postgresql.replication.fluent.logical.ChainedLogicalStreamBuilder	optionsWithMetadata(org.postgresql.replication.fluent.logical.ChainedLogicalStreamBuilder builder) Allows MessageDecoder to configure options with which the replication stream is started.
org.postgresql.replication.fluent.logical.ChainedLogicalStreamBuilder	optionsWithoutMetadata(org.postgresql.replication.fluent.logical.ChainedLogicalStreamBuilder builder) Allows MessageDecoder to configure options with which the replication stream is started.
protected void	outOfOrderChunk(byte[] content) This method is called when a database server or Debezium crashes.
void	processNotEmptyMessage(ByteBuffer buffer, ReplicationStream.ReplicationMessageProcessor processor, TypeRegistry typeRegistry)
private void	replaceFirstNonWhiteChar(byte[] array, byte to)
void	setContainsMetadata(boolean containsMetadata) Signals to this decoder whether messages contain type metadata or not.

Methods inherited from class io.debezium.connector.postgresql.connection.AbstractMessageDecoder

processMessage, shouldMessageBeSkipped

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOGGER

private static final org.slf4j.Logger LOGGER

TAB

private static final byte TAB

See Also:

Constant Field Values

CR

private static final byte CR

See Also:

Constant Field Values

SPACE

private static final byte SPACE

See Also:

Constant Field Values

COMMA

private static final byte COMMA

See Also:

Constant Field Values

RIGHT_BRACKET

private static final byte RIGHT_BRACKET

See Also:

Constant Field Values

LEFT_BRACE

private static final byte LEFT_BRACE

See Also:

Constant Field Values

RIGHT_BRACE

private static final byte RIGHT_BRACE

See Also:

Constant Field Values

UNDEFINED_LONG

private static final long UNDEFINED_LONG

See Also:

Constant Field Values

dateTime

private final DateTimeFormat dateTime

containsMetadata

private boolean containsMetadata

messageInProgress

private boolean messageInProgress

currentChunk

private byte[] currentChunk

To identify if the last current chunk is the last one we can send the current one for processing only after we read the next one or the end of message fragment.

txId

private long txId

commitTime

private Instant commitTime

Constructor Detail

StreamingWal2JsonMessageDecoder

public StreamingWal2JsonMessageDecoder(MessageDecoderConfig config)

Method Detail

processNotEmptyMessage

public void processNotEmptyMessage(ByteBuffer buffer,
                                   ReplicationStream.ReplicationMessageProcessor processor,
                                   TypeRegistry typeRegistry)
                            throws SQLException,
                                   InterruptedException

Specified by:

processNotEmptyMessage in class AbstractMessageDecoder

Throws:

SQLException

InterruptedException

nonInitialChunk

protected void nonInitialChunk(ReplicationStream.ReplicationMessageProcessor processor,
                               TypeRegistry typeRegistry,
                               byte[] content)
                        throws IOException,
                               SQLException,
                               InterruptedException

Throws:

IOException

SQLException

InterruptedException

outOfOrderChunk

protected void outOfOrderChunk(byte[] content)

This method is called when a database server or Debezium crashes. With wal2json streaming mode it can happen that the message preamble is no replayed but the message is streamed form the middle. This issue is very hard to reproduce so a precaution is taken and metadata are filled with synthetic values.

The new wal2json format will be resilient to this situation.

Parameters:

content -

getLastNonWhiteChar

private byte getLastNonWhiteChar(byte[] array)
                          throws IllegalArgumentException

Throws:

IllegalArgumentException

getFirstNonWhiteChar

private byte getFirstNonWhiteChar(byte[] array)
                           throws IllegalArgumentException

Throws:

IllegalArgumentException

replaceFirstNonWhiteChar

private void replaceFirstNonWhiteChar(byte[] array,
                                      byte to)

isWhitespace

private boolean isWhitespace(byte c)

doProcessMessage

private void doProcessMessage(ReplicationStream.ReplicationMessageProcessor processor,
                              TypeRegistry typeRegistry,
                              byte[] content,
                              boolean lastMessage)
                       throws IOException,
                              SQLException,
                              InterruptedException

Throws:

IOException

SQLException

InterruptedException

optionsWithMetadata

public org.postgresql.replication.fluent.logical.ChainedLogicalStreamBuilder optionsWithMetadata(org.postgresql.replication.fluent.logical.ChainedLogicalStreamBuilder builder)

Description copied from interface: MessageDecoder

Allows MessageDecoder to configure options with which the replication stream is started. The messages CAN contain type metadata. See PostgreSQL command START_REPLICATION SLOT for more details.

Returns:

the builder instance

optionsWithoutMetadata

public org.postgresql.replication.fluent.logical.ChainedLogicalStreamBuilder optionsWithoutMetadata(org.postgresql.replication.fluent.logical.ChainedLogicalStreamBuilder builder)

Description copied from interface: MessageDecoder

Allows MessageDecoder to configure options with which the replication stream is started. The messages MUST NOT contain type metadata. See PostgreSQL command START_REPLICATION SLOT for more details.

Returns:

the builder instance

setContainsMetadata

public void setContainsMetadata(boolean containsMetadata)

Description copied from interface: MessageDecoder

Signals to this decoder whether messages contain type metadata or not.