Package com.snowflake.kafka.connector

Class SnowflakeSinkConnector

  • All Implemented Interfaces:
    org.apache.kafka.connect.components.Versioned
    public class SnowflakeSinkConnector
extends org.apache.kafka.connect.sink.SinkConnector
    SnowflakeSinkConnector implements SinkConnector for Kafka Connect framework.

    Expected configuration: including topic names, partition numbers, snowflake connection info and credentials info

    Creates snowflake internal stages, snowflake tables provides configuration to SinkTasks running on Kafka Connect Workers.

    • Field Summary

      • Fields inherited from class org.apache.kafka.connect.sink.SinkConnector

        TOPICS_CONFIG

      • Fields inherited from class org.apache.kafka.connect.connector.Connector

        context

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      org.apache.kafka.common.config.ConfigDef config()  
      void start​(Map<String,​String> parsedConfig)
      start method will only be called on a clean connector, i.e.
      void stop()
      Stop method will be called to stop a connector, cleans up snowflake internal stages, after making sure that there are no pending files to ingest.
      Class<? extends org.apache.kafka.connect.connector.Task> taskClass()  
      List<Map<String,​String>> taskConfigs​(int maxTasks)
      taskConfigs method returns a set of configurations for SinkTasks based on the current configuration, producing at most 'maxTasks' configurations
      org.apache.kafka.common.config.Config validate​(Map<String,​String> connectorConfigs)  
      String version()  

      • Methods inherited from class org.apache.kafka.connect.sink.SinkConnector

        context

      • Methods inherited from class org.apache.kafka.connect.connector.Connector

        initialize, initialize, reconfigure

    • Constructor Detail

      • SnowflakeSinkConnector

        public SnowflakeSinkConnector()
        No-Arg constructor. Required by Kafka Connect framework

    • Method Detail

      • start

        public void start​(Map<String,​String> parsedConfig)
        start method will only be called on a clean connector, i.e. it has either just been instantiated and initialized or stop () has been invoked. loads configuration and validates.

        Creates snowflake internal stages and snowflake tables

        Specified by:
        start in class org.apache.kafka.connect.connector.Connector
        Parameters:
        parsedConfig - has the configuration settings

      • stop

        public void stop()
        Stop method will be called to stop a connector, cleans up snowflake internal stages, after making sure that there are no pending files to ingest.

        Cleans up pipes, after making sure there are no pending files to ingest.

        Also ensures that there are no leaked stages, no leaked staged files, and no leaked pipes

        Specified by:
        stop in class org.apache.kafka.connect.connector.Connector

      • taskClass

        public Class<? extends org.apache.kafka.connect.connector.Task> taskClass()
        Specified by:
        taskClass in class org.apache.kafka.connect.connector.Connector
        Returns:
        Sink task class

      • taskConfigs

        public List<Map<String,​String>> taskConfigs​(int maxTasks)
        taskConfigs method returns a set of configurations for SinkTasks based on the current configuration, producing at most 'maxTasks' configurations
        Specified by:
        taskConfigs in class org.apache.kafka.connect.connector.Connector
        Parameters:
        maxTasks - maximum number of SinkTasks for this instance of SnowflakeSinkConnector
        Returns:
        a list containing 'maxTasks' copies of the configuration

      • config

        public org.apache.kafka.common.config.ConfigDef config()
        Specified by:
        config in class org.apache.kafka.connect.connector.Connector
        Returns:
        ConfigDef with original configuration properties

      • validate

        public org.apache.kafka.common.config.Config validate​(Map<String,​String> connectorConfigs)
        Overrides:
        validate in class org.apache.kafka.connect.connector.Connector

      • version

        public String version()
        Returns:
        connector version