Package com.snowflake.kafka.connector.internal

Interface SnowflakeSinkService

  • All Known Implementing Classes:
    SnowflakeSinkServiceV2
    public interface SnowflakeSinkService
    Background service of data sink, responsible to create/drop pipe and ingest/purge files

    • Method Detail

      • startPartition

        void startPartition​(String tableName,
                    org.apache.kafka.common.TopicPartition topicPartition)
        Start the Task. This should handle any configuration parsing and one-time setup of the task.
        Parameters:
        tableName - destination table name
        topicPartition - TopicPartition passed from Kafka

      • startPartitions

        void startPartitions​(Collection<org.apache.kafka.common.TopicPartition> partitions,
                     Map<String,​String> topic2Table)
        Start a collection of TopicPartition. This should handle any configuration parsing and one-time setup of the task.
        Parameters:
        partitions - collection of topic partitions
        topic2Table - a mapping from topic to table

      • insert

        void insert​(Collection<org.apache.kafka.connect.sink.SinkRecord> records)
        call pipe to insert a collections of JSON records will trigger time based flush
        Parameters:
        records - record content

      • insert

        void insert​(org.apache.kafka.connect.sink.SinkRecord record)
        call pipe to insert a JSON record will not trigger time based flush
        Parameters:
        record - record content

      • getOffset

        long getOffset​(org.apache.kafka.common.TopicPartition topicPartition)
        retrieve offset of last loaded record for given pipe name
        Parameters:
        topicPartition - topic and partition
        Returns:
        offset, or -1 for empty

      • getPartitionCount

        int getPartitionCount()
        get the number of partitions assigned to this sink service
        Returns:
        number of partitions

      • callAllGetOffset

        void callAllGetOffset()
        used for testing only

      • closeAll

        void closeAll()
        terminate all tasks and close this service instance

      • close

        void close​(Collection<org.apache.kafka.common.TopicPartition> partitions)
        terminate given topic partitions
        Parameters:
        partitions - a list of topic partition

      • setIsStoppedToTrue

        void setIsStoppedToTrue()
        close all cleaner thread but have no effect on sink service context

      • isClosed

        boolean isClosed()
        retrieve sink service status
        Returns:
        true is closed

      • setRecordNumber

        void setRecordNumber​(long num)
        change maximum number of record cached in buffer to control the flush rate, 0 for unlimited
        Parameters:
        num - a non negative long number represents number of record limitation

      • setFileSize

        void setFileSize​(long size)
        change data size of buffer to control the flush rate, the minimum file size is controlled by SnowflakeSinkConnectorConfig.BUFFER_SIZE_BYTES_MIN

        Please note: The buffer size for Streaming and snowpipe doesnt necessarily translate to same file size in Snowflake.

        There is Java to UTF conversion followed by file compression in gzip.

        Parameters:
        size - a non negative long number represents data size limitation

      • setTopic2TableMap

        void setTopic2TableMap​(Map<String,​String> topic2TableMap)
        pass topic to table map to sink service
        Parameters:
        topic2TableMap - a String to String Map represents topic to table map

      • setMetadataConfig

        void setMetadataConfig​(SnowflakeMetadataConfig configMap)
        set the metadata config to let user control what metadata to be collected into SF db
        Parameters:
        configMap - a String to String Map

      • getRecordNumber

        long getRecordNumber()
        Returns:
        current number of record limitation

      • getFlushTime

        long getFlushTime()
        Returns:
        current flush time in seconds

      • getFileSize

        long getFileSize()
        Returns:
        current file size limitation

      • setCustomJMXMetrics

        void setCustomJMXMetrics​(boolean enableJMX)

      • setSinkTaskContext

        default void setSinkTaskContext​(org.apache.kafka.connect.sink.SinkTaskContext sinkTaskContext)

      • getMetricRegistry

        Optional<com.codahale.metrics.MetricRegistry> getMetricRegistry​(String partitionIdentifier)