Package com.snowflake.kafka.connector.records

Class RecordService

  • public class RecordService
extends Object

    • Constructor Detail

      • RecordService

        public RecordService​(SnowflakeTelemetryService telemetryService)
        process records output JSON format: { "meta": { "offset": 123, "topic": "topic name", "partition": 123, "key":"key name" } "content": "record content" }

        create a JsonRecordService instance

        Parameters:
        telemetryService - Telemetry Service Instance. Can be null.

      • RecordService

        public RecordService()
        Record service with null telemetry Service, only use it for testing.

    • Method Detail

      • setAndGetEnableSchematizationFromConfig

        public boolean setAndGetEnableSchematizationFromConfig​(Map<String,​String> connectorConfig)
        extract enableSchematization from the connector config and set the value for the recordService

        The extracted boolean is returned for external usage.

        Parameters:
        connectorConfig - the connector config map
        Returns:
        a boolean indicating whether schematization is enabled

      • setEnableSchematization

        public void setEnableSchematization​(boolean enableSchematization)
        Directly set the enableSchematization through param

        This method is only for testing

        Parameters:
        enableSchematization - whether we should enable schematization or not

      • setBehaviorOnNullValues

        public void setBehaviorOnNullValues​(SnowflakeSinkConnectorConfig.BehaviorOnNullValues behaviorOnNullValues)
        Directly set the behaviorOnNullValues through param

        This method is only for testing

        Parameters:
        behaviorOnNullValues - how to handle null values

      • getProcessedRecordForSnowpipe

        public String getProcessedRecordForSnowpipe​(org.apache.kafka.connect.sink.SinkRecord record)
        Given a single Record from put API, process it and convert it into a Json String.

        Remember, Snowflake table has two columns, both of them are VARIANT columns whose contents are in JSON

        Parameters:
        record - record from Kafka
        Returns:
        Json String with metadata and actual Payload from Kafka Record

      • getProcessedRecordForStreamingIngest

        public Map<String,​Object> getProcessedRecordForStreamingIngest​(org.apache.kafka.connect.sink.SinkRecord record)
                                                              throws net.snowflake.client.jdbc.internal.fasterxml.jackson.core.JsonProcessingException
        Given a single Record from put API, process it and convert it into Map of String and Object.

        This map contains two Keys and its corresponding values

        Two keys are the column names and values are its contents.

        Remember, Snowflake table has two columns, both of them are VARIANT columns whose contents are in JSON

        When schematization is enabled, the content of the record is extracted into a map

        Parameters:
        record - record from Kafka to (Which was serialized in Json)
        Returns:
        Json String with metadata and actual Payload from Kafka Record
        Throws:
        net.snowflake.client.jdbc.internal.fasterxml.jackson.core.JsonProcessingException

      • convertToJson

        public static net.snowflake.client.jdbc.internal.fasterxml.jackson.databind.JsonNode convertToJson​(org.apache.kafka.connect.data.Schema schema,
                                                                                                   Object logicalValue)
        Convert this object, in the org.apache.kafka.connect.data format, into a JSON object, returning the converted object.
        Parameters:
        schema - schema of the object
        logicalValue - object to be converted
        Returns:
        a JsonNode of the object

      • shouldSkipNullValue

        public boolean shouldSkipNullValue​(org.apache.kafka.connect.sink.SinkRecord record,
                                   SnowflakeSinkConnectorConfig.BehaviorOnNullValues behaviorOnNullValues)
        Returns true if we want to skip this record since the value is null or it is an empty json string.

        Remember, we need to check what is the value schema. Depending on the value schema, we need to find out if the value is null or empty JSON. It can be empty JSON string in case of custom snowflake converters.

        If the value is an empty JSON node, we could assume the value passed was null.

        Parameters:
        record - record sent from Kafka to KC
        behaviorOnNullValues - behavior passed inside KC
        Returns:
        true if we would skip adding it to buffer
        See Also:
        when bytes == null case