Package com.snowflake.kafka.connector

Class SnowflakeSinkTask

java.lang.Object

org.apache.kafka.connect.sink.SinkTask

com.snowflake.kafka.connector.SnowflakeSinkTask

All Implemented Interfaces:

org.apache.kafka.connect.connector.Task

public class SnowflakeSinkTask
extends org.apache.kafka.connect.sink.SinkTask

SnowflakeSinkTask implements SinkTask for Kafka Connect framework.

Expects configuration from SnowflakeSinkConnector

Creates sink service instance, takes records loaded from those Kafka partitions and ingests to Snowflake via Sink service

Field Summary

Fields inherited from class org.apache.kafka.connect.sink.SinkTask

context, TOPICS_CONFIG, TOPICS_REGEX_CONFIG

Constructor Summary

Constructors

Constructor	Description
SnowflakeSinkTask()	default constructor, invoked by kafka connect framework
SnowflakeSinkTask(SnowflakeSinkService service, SnowflakeConnectionService connectionService)
SnowflakeSinkTask(SnowflakeSinkService service, SnowflakeConnectionService connectionService, Map<String,String> topic2table)

Method Summary

Modifier and Type	Method	Description
void	close(Collection<org.apache.kafka.common.TopicPartition> partitions)	Closes sink service
Optional<SnowflakeConnectionService>	getSnowflakeConnection()	Return an instance of SnowflakeConnection if it was set previously by calling Start().
void	open(Collection<org.apache.kafka.common.TopicPartition> partitions)	init ingestion task in Sink service
Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata>	preCommit(Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata> offsets)	Sync committed offsets
void	put(Collection<org.apache.kafka.connect.sink.SinkRecord> records)	ingest records to Snowflake
void	start(Map<String,String> parsedConfig)	start method handles configuration parsing and one-time setup of the task.
void	stop()	stop method is invoked only once outstanding calls to other methods have completed.
String	version()

Methods inherited from class org.apache.kafka.connect.sink.SinkTask

flush, initialize, onPartitionsAssigned, onPartitionsRevoked

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SnowflakeSinkTask

public SnowflakeSinkTask()

default constructor, invoked by kafka connect framework

SnowflakeSinkTask

public SnowflakeSinkTask(SnowflakeSinkService service,
                         SnowflakeConnectionService connectionService)

SnowflakeSinkTask

public SnowflakeSinkTask(SnowflakeSinkService service,
                         SnowflakeConnectionService connectionService,
                         Map<String,String> topic2table)

Method Detail

getSnowflakeConnection

public Optional<SnowflakeConnectionService> getSnowflakeConnection()

Return an instance of SnowflakeConnection if it was set previously by calling Start(). Else, return an empty

Returns:

Optional of SnowflakeConnectionService

start

public void start(Map<String,String> parsedConfig)

start method handles configuration parsing and one-time setup of the task. loads configuration

Specified by:

start in interface org.apache.kafka.connect.connector.Task

Specified by:

start in class org.apache.kafka.connect.sink.SinkTask

Parameters:

parsedConfig - - has the configuration settings

stop

public void stop()

stop method is invoked only once outstanding calls to other methods have completed. e.g. after current put, and a final preCommit has completed.

Specified by:

stop in interface org.apache.kafka.connect.connector.Task

Specified by:

stop in class org.apache.kafka.connect.sink.SinkTask

open

public void open(Collection<org.apache.kafka.common.TopicPartition> partitions)

init ingestion task in Sink service

Overrides:

open in class org.apache.kafka.connect.sink.SinkTask

Parameters:

partitions - - The list of all partitions that are now assigned to the task

close

public void close(Collection<org.apache.kafka.common.TopicPartition> partitions)

Closes sink service

Closes all running task because the parameter of open function contains all partition info but not only the new partition

Overrides:

close in class org.apache.kafka.connect.sink.SinkTask

Parameters:

partitions - - The list of all partitions that were assigned to the task

put

public void put(Collection<org.apache.kafka.connect.sink.SinkRecord> records)

ingest records to Snowflake

Specified by:

put in class org.apache.kafka.connect.sink.SinkTask

Parameters:

records - - collection of records from kafka topic/partitions for this connector

preCommit

public Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata> preCommit(Map<org.apache.kafka.common.TopicPartition,org.apache.kafka.clients.consumer.OffsetAndMetadata> offsets)
                                                                                                                throws org.apache.kafka.connect.errors.RetriableException

Sync committed offsets

Overrides:

preCommit in class org.apache.kafka.connect.sink.SinkTask

Parameters:

offsets - - the current map of offsets as of the last call to put

Returns:

an empty map if Connect-managed offset commit is not desired, otherwise a map of offsets by topic-partition that are safe to commit. If we return the same offsets that was passed in, Kafka Connect assumes that all offsets that are already passed to put() are safe to commit.

Throws:

org.apache.kafka.connect.errors.RetriableException - when meet any issue during processing

version

public String version()

Returns:

connector version