Package com.snowflake.kafka.connector.internal.streaming

Class TopicPartitionChannel.StreamingBuffer

java.lang.Object

com.snowflake.kafka.connector.internal.PartitionBuffer<net.snowflake.ingest.utils.Pair<List<Map<String,Object>>,List<Long>>>

com.snowflake.kafka.connector.internal.streaming.TopicPartitionChannel.StreamingBuffer

Enclosing class:

TopicPartitionChannel

protected class TopicPartitionChannel.StreamingBuffer
extends PartitionBuffer<net.snowflake.ingest.utils.Pair<List<Map<String,Object>>,List<Long>>>

A buffer which holds the rows before calling insertRows API. It implements the PartitionBuffer class which has all common fields about a buffer.

This buffer is a bit different from Snowpipe Buffer. In StreamingBuffer we buffer incoming records from Kafka and once threshold has reached, we would call insertRows API to insert into Snowflake.

We would transform kafka records to Snowflake understood records (In JSON format) just before calling insertRows API.

Method Summary

Modifier and Type	Method	Description
net.snowflake.ingest.utils.Pair<List<Map<String,Object>>,List<Long>>	getData()	Get all rows and their offsets.
org.apache.kafka.connect.sink.SinkRecord	getSinkRecord(long idx)
List<org.apache.kafka.connect.sink.SinkRecord>	getSinkRecords()	TODO:SNOW-552576 Avoid extra memory in buffer.
void	insert(org.apache.kafka.connect.sink.SinkRecord kafkaSinkRecord)	Inserts the row into Buffer.

Methods inherited from class com.snowflake.kafka.connector.internal.PartitionBuffer

getBufferSizeBytes, getFirstOffset, getLastOffset, getNumOfRecords, isEmpty, setBufferSizeBytes, setFirstOffset, setLastOffset, setNumOfRecords, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Method Detail

insert

public void insert(org.apache.kafka.connect.sink.SinkRecord kafkaSinkRecord)

Description copied from class: PartitionBuffer

Inserts the row into Buffer.

Specified by:

insert in class PartitionBuffer<net.snowflake.ingest.utils.Pair<List<Map<String,Object>>,List<Long>>>

Parameters:

kafkaSinkRecord - the record which kafka sends to KC. We also convert the record to a format which is understood by Snowflake Table

getData

public net.snowflake.ingest.utils.Pair<List<Map<String,Object>>,List<Long>> getData()

Get all rows and their offsets. Each map corresponds to one row whose keys are column names and values are corresponding data in that column.

This goes over through all buffered kafka records and transforms into JsonSchema and JsonNode Check TopicPartitionChannel.handleNativeRecord(SinkRecord, boolean)

Specified by:

getData in class PartitionBuffer<net.snowflake.ingest.utils.Pair<List<Map<String,Object>>,List<Long>>>

Returns:

A pair that contains the records and their corresponding offsets

getSinkRecords

public List<org.apache.kafka.connect.sink.SinkRecord> getSinkRecords()

Description copied from class: PartitionBuffer

TODO:SNOW-552576 Avoid extra memory in buffer.

Specified by:

getSinkRecords in class PartitionBuffer<net.snowflake.ingest.utils.Pair<List<Map<String,Object>>,List<Long>>>

Returns:

the sinkrecords corresponding to this buffer

getSinkRecord

public org.apache.kafka.connect.sink.SinkRecord getSinkRecord(long idx)