Package com.snowflake.kafka.connector.internal.streaming

Class SnowflakeSinkServiceV2

java.lang.Object

com.snowflake.kafka.connector.internal.streaming.SnowflakeSinkServiceV2

All Implemented Interfaces:

SnowflakeSinkService

public class SnowflakeSinkServiceV2
extends Object
implements SnowflakeSinkService

This is per task configuration. A task can be assigned multiple partitions. Major methods are startTask, insert, getOffset and close methods.

StartTask: Called when partitions are assigned. Responsible for generating the POJOs.

Insert and getOffset are called when SnowflakeSinkTask.put(Collection) and SnowflakeSinkTask.preCommit(Map) APIs are called.

This implementation of SinkService uses Streaming Snowpipe (Streaming Ingestion)

Hence this initializes the channel, opens, closes. The StreamingIngestChannel resides inside TopicPartitionChannel which is per partition.

Constructor Summary

Constructors

Constructor	Description
SnowflakeSinkServiceV2(long flushTimeSeconds, long fileSizeBytes, long recordNum, SnowflakeConnectionService conn, RecordService recordService, SnowflakeTelemetryService telemetryService, Map<String,String> topicToTableMap, SnowflakeSinkConnectorConfig.BehaviorOnNullValues behaviorOnNullValues, boolean enableCustomJMXMonitoring, KafkaRecordErrorReporter kafkaRecordErrorReporter, org.apache.kafka.connect.sink.SinkTaskContext sinkTaskContext, net.snowflake.ingest.streaming.SnowflakeStreamingIngestClient streamingIngestClient, Map<String,String> connectorConfig, boolean enableSchematization, Map<String,TopicPartitionChannel> partitionsToChannel)
SnowflakeSinkServiceV2(SnowflakeConnectionService conn, Map<String,String> connectorConfig)

Method Summary

Modifier and Type	Method	Description
void	callAllGetOffset()	used for testing only
void	close(Collection<org.apache.kafka.common.TopicPartition> partitions)	This function is called during rebalance.
void	closeAll()	terminate all tasks and close this service instance
SnowflakeSinkConnectorConfig.BehaviorOnNullValues	getBehaviorOnNullValuesConfig()
long	getFileSize()	This is more of size in bytes of buffered records.
long	getFlushTime()
Optional<com.codahale.metrics.MetricRegistry>	getMetricRegistry(String partitionChannelKey)
long	getOffset(org.apache.kafka.common.TopicPartition topicPartition)	retrieve offset of last loaded record for given pipe name
int	getPartitionCount()	get the number of partitions assigned to this sink service
long	getRecordNumber()
net.snowflake.ingest.streaming.SnowflakeStreamingIngestClient	getStreamingIngestClient()
protected Optional<TopicPartitionChannel>	getTopicPartitionChannelFromCacheKey(String topicPartitionChannelKey)	Used for testing Only
void	insert(Collection<org.apache.kafka.connect.sink.SinkRecord> records)	Inserts the given record into buffer and then eventually calls insertRows API if buffer threshold has reached.
void	insert(org.apache.kafka.connect.sink.SinkRecord record)	Inserts individual records into buffer.
boolean	isClosed()	retrieve sink service status
static String	partitionChannelKey(String topic, int partition)	Gets a unique identifier consisting of topic name and partition number.
void	setBehaviorOnNullValuesConfig(SnowflakeSinkConnectorConfig.BehaviorOnNullValues behavior)
void	setCustomJMXMetrics(boolean enableJMX)
void	setErrorReporter(KafkaRecordErrorReporter kafkaRecordErrorReporter)
void	setFileSize(long size)	Assume this is buffer size in bytes, since this is streaming ingestion
void	setFlushTime(long time)	change flush rate of sink service the minimum flush time is controlled by SnowflakeSinkConnectorConfig.BUFFER_FLUSH_TIME_SEC_MIN
void	setIsStoppedToTrue()	close all cleaner thread but have no effect on sink service context
void	setMetadataConfig(SnowflakeMetadataConfig configMap)	set the metadata config to let user control what metadata to be collected into SF db
void	setRecordNumber(long num)	change maximum number of record cached in buffer to control the flush rate, 0 for unlimited
void	setSinkTaskContext(org.apache.kafka.connect.sink.SinkTaskContext sinkTaskContext)
void	setTopic2TableMap(Map<String,String> topicToTableMap)	pass topic to table map to sink service
void	startPartition(String tableName, org.apache.kafka.common.TopicPartition topicPartition)	Creates a table if it doesnt exist in Snowflake.
void	startPartitions(Collection<org.apache.kafka.common.TopicPartition> partitions, Map<String,String> topic2Table)	Initializes multiple Channels and partitionsToChannel maps with new instances of TopicPartitionChannel

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SnowflakeSinkServiceV2

public SnowflakeSinkServiceV2(SnowflakeConnectionService conn,
                              Map<String,String> connectorConfig)

SnowflakeSinkServiceV2

public SnowflakeSinkServiceV2(long flushTimeSeconds,
                              long fileSizeBytes,
                              long recordNum,
                              SnowflakeConnectionService conn,
                              RecordService recordService,
                              SnowflakeTelemetryService telemetryService,
                              Map<String,String> topicToTableMap,
                              SnowflakeSinkConnectorConfig.BehaviorOnNullValues behaviorOnNullValues,
                              boolean enableCustomJMXMonitoring,
                              KafkaRecordErrorReporter kafkaRecordErrorReporter,
                              org.apache.kafka.connect.sink.SinkTaskContext sinkTaskContext,
                              net.snowflake.ingest.streaming.SnowflakeStreamingIngestClient streamingIngestClient,
                              Map<String,String> connectorConfig,
                              boolean enableSchematization,
                              Map<String,TopicPartitionChannel> partitionsToChannel)

Method Detail

startPartition

public void startPartition(String tableName,
                           org.apache.kafka.common.TopicPartition topicPartition)

Creates a table if it doesnt exist in Snowflake.

Initializes the Channel and partitionsToChannel map with new instance of TopicPartitionChannel

Specified by:

startPartition in interface SnowflakeSinkService

Parameters:

tableName - destination table name

topicPartition - TopicPartition passed from Kafka

startPartitions

public void startPartitions(Collection<org.apache.kafka.common.TopicPartition> partitions,
                            Map<String,String> topic2Table)

Initializes multiple Channels and partitionsToChannel maps with new instances of TopicPartitionChannel

Specified by:

startPartitions in interface SnowflakeSinkService

Parameters:

partitions - collection of topic partition

topic2Table - map of topic to table name

insert

public void insert(Collection<org.apache.kafka.connect.sink.SinkRecord> records)

Inserts the given record into buffer and then eventually calls insertRows API if buffer threshold has reached.

TODO: SNOW-473896 - Please note we will get away with Buffering logic in future commits.

Specified by:

insert in interface SnowflakeSinkService

Parameters:

records - records coming from Kafka. Please note, they are not just from single topic and partition. It depends on the kafka connect worker node which can consume from multiple Topic and multiple Partitions

insert

public void insert(org.apache.kafka.connect.sink.SinkRecord record)

Inserts individual records into buffer. It fetches the TopicPartitionChannel from the map and then each partition(Streaming channel) calls its respective insertRows API

Specified by:

insert in interface SnowflakeSinkService

Parameters:

record - record content

getOffset

public long getOffset(org.apache.kafka.common.TopicPartition topicPartition)

Description copied from interface: SnowflakeSinkService

retrieve offset of last loaded record for given pipe name

Specified by:

getOffset in interface SnowflakeSinkService

Parameters:

topicPartition - topic and partition

Returns:

offset, or -1 for empty

getPartitionCount

public int getPartitionCount()

Description copied from interface: SnowflakeSinkService

get the number of partitions assigned to this sink service

Specified by:

getPartitionCount in interface SnowflakeSinkService

Returns:

number of partitions

callAllGetOffset

public void callAllGetOffset()

Description copied from interface: SnowflakeSinkService

used for testing only

Specified by:

callAllGetOffset in interface SnowflakeSinkService

closeAll

public void closeAll()

Description copied from interface: SnowflakeSinkService

terminate all tasks and close this service instance

Specified by:

closeAll in interface SnowflakeSinkService

close

public void close(Collection<org.apache.kafka.common.TopicPartition> partitions)

This function is called during rebalance.

All the channels are closed. The client is still active. Upon rebalance, (inside SnowflakeSinkTask.open(Collection) we will reopen the channel.

We will wipe the cache partitionsToChannel so that in SnowflakeSinkTask.open(Collection) we reinstantiate and fetch offsetToken

Specified by:

close in interface SnowflakeSinkService

Parameters:

partitions - a list of topic partition

setIsStoppedToTrue

public void setIsStoppedToTrue()

Description copied from interface: SnowflakeSinkService

close all cleaner thread but have no effect on sink service context

Specified by:

setIsStoppedToTrue in interface SnowflakeSinkService

isClosed

public boolean isClosed()

Description copied from interface: SnowflakeSinkService

retrieve sink service status

Specified by:

isClosed in interface SnowflakeSinkService

Returns:

true is closed

setRecordNumber

public void setRecordNumber(long num)

Description copied from interface: SnowflakeSinkService

change maximum number of record cached in buffer to control the flush rate, 0 for unlimited

Specified by:

setRecordNumber in interface SnowflakeSinkService

Parameters:

num - a non negative long number represents number of record limitation

setFileSize

public void setFileSize(long size)

Assume this is buffer size in bytes, since this is streaming ingestion

Specified by:

setFileSize in interface SnowflakeSinkService

Parameters:

size - in bytes - a non negative long number representing size of internal buffer for flush.

setTopic2TableMap

public void setTopic2TableMap(Map<String,String> topicToTableMap)

Description copied from interface: SnowflakeSinkService

pass topic to table map to sink service

Specified by:

setTopic2TableMap in interface SnowflakeSinkService

Parameters:

topicToTableMap - a String to String Map represents topic to table map

setFlushTime

public void setFlushTime(long time)

Description copied from interface: SnowflakeSinkService

change flush rate of sink service the minimum flush time is controlled by SnowflakeSinkConnectorConfig.BUFFER_FLUSH_TIME_SEC_MIN

Specified by:

setFlushTime in interface SnowflakeSinkService

Parameters:

time - a non negative long number represents service flush time in seconds

setMetadataConfig

public void setMetadataConfig(SnowflakeMetadataConfig configMap)

Description copied from interface: SnowflakeSinkService

set the metadata config to let user control what metadata to be collected into SF db

Specified by:

setMetadataConfig in interface SnowflakeSinkService

Parameters:

configMap - a String to String Map

getRecordNumber

public long getRecordNumber()

Specified by:

getRecordNumber in interface SnowflakeSinkService

Returns:

current number of record limitation

getFlushTime

public long getFlushTime()

Specified by:

getFlushTime in interface SnowflakeSinkService

Returns:

current flush time in seconds

getFileSize

public long getFileSize()

This is more of size in bytes of buffered records. This necessarily doesnt translates to files created by Streaming Ingest since they are compressed. So there is no 1:1 mapping.

Specified by:

getFileSize in interface SnowflakeSinkService

Returns:

current file size limitation

setBehaviorOnNullValuesConfig

public void setBehaviorOnNullValuesConfig(SnowflakeSinkConnectorConfig.BehaviorOnNullValues behavior)

Specified by:

setBehaviorOnNullValuesConfig in interface SnowflakeSinkService

setCustomJMXMetrics

public void setCustomJMXMetrics(boolean enableJMX)

Specified by:

setCustomJMXMetrics in interface SnowflakeSinkService

getBehaviorOnNullValuesConfig

public SnowflakeSinkConnectorConfig.BehaviorOnNullValues getBehaviorOnNullValuesConfig()

Specified by:

getBehaviorOnNullValuesConfig in interface SnowflakeSinkService

setErrorReporter

public void setErrorReporter(KafkaRecordErrorReporter kafkaRecordErrorReporter)

Specified by:

setErrorReporter in interface SnowflakeSinkService

setSinkTaskContext

public void setSinkTaskContext(org.apache.kafka.connect.sink.SinkTaskContext sinkTaskContext)

Specified by:

setSinkTaskContext in interface SnowflakeSinkService

getMetricRegistry

public Optional<com.codahale.metrics.MetricRegistry> getMetricRegistry(String partitionChannelKey)

Specified by:

getMetricRegistry in interface SnowflakeSinkService

partitionChannelKey

public static String partitionChannelKey(String topic,
                                         int partition)

Gets a unique identifier consisting of topic name and partition number.

Parameters:

topic - topic name

partition - partition number

Returns:

combinartion of topic and partition

getStreamingIngestClient

public net.snowflake.ingest.streaming.SnowflakeStreamingIngestClient getStreamingIngestClient()

getTopicPartitionChannelFromCacheKey

protected Optional<TopicPartitionChannel> getTopicPartitionChannelFromCacheKey(String topicPartitionChannelKey)

Used for testing Only

Parameters:

topicPartitionChannelKey - look partitionChannelKey(String, int) for key format

Returns:

TopicPartitionChannel if present in partitionsToChannel Map else null