Package com.snowflake.kafka.connector.internal.telemetry

Class SnowflakeTelemetryPipeStatus

    • Constructor Detail

      • SnowflakeTelemetryPipeStatus

        public SnowflakeTelemetryPipeStatus​(String tableName,
                                    String stageName,
                                    String pipeName,
                                    boolean enableCustomJMXConfig,
                                    MetricsJmxReporter metricsJmxReporter)

    • Method Detail

      • updateKafkaLag

        public void updateKafkaLag​(long lag)
        Kafka Lag is time between kafka connector sees the record and time record was inserted into kafka.

        Check the implementation of updateLag on what is done with this lag.

        Parameters:
        lag -

      • updateIngestionLag

        public void updateIngestionLag​(long lag)
        Ingestion Lag is time between kafka connector flushes the file and time file was first found from insertReport/loadHistory API.

        Check the implementation of updateLag on what is done with this lag.

        Parameters:
        lag -

      • updateCommitLag

        public void updateCommitLag​(long lag)
        Commit Lag is time between kafka connector commits the after calling insertFiles API and time file was flushed into internal stage.

        Check the implementation of updateLag on what is done with this lag.

        Parameters:
        lag -

      • updateBrokenRecordMetrics

        public void updateBrokenRecordMetrics​(long n)
        When either key or value is broken.
        Parameters:
        n - number of records

      • updateFailedIngestionMetrics

        public void updateFailedIngestionMetrics​(long n)
        When Ingestion status of n number of files is not found/failed.
        Parameters:
        n - number of files failed ingestion

      • isEmpty

        public boolean isEmpty()
        Specified by:
        isEmpty in class SnowflakeTelemetryBasicInfo
        Returns:
        true if it would suggest that their was no update to corresponding implementation's member variables. Or, in other words, the corresponding partition didnt receive any records, in which case we would not call telemetry API.

      • dumpTo

        public void dumpTo​(net.snowflake.client.jdbc.internal.fasterxml.jackson.databind.node.ObjectNode msg)
        Description copied from class: SnowflakeTelemetryBasicInfo
        Adds the required fields into the given ObjectNode which will then be used as payload in Telemetry API
        Specified by:
        dumpTo in class SnowflakeTelemetryBasicInfo
        Parameters:
        msg - ObjectNode in which extra fields needs to be added.

      • setProcessedOffset

        public void setProcessedOffset​(long processedOffset)

      • setFlushedOffset

        public void setFlushedOffset​(long flushedOffset)

      • setCommittedOffset

        public void setCommittedOffset​(long committedOffset)

      • setPurgedOffsetAtomically

        public void setPurgedOffsetAtomically​(LongUnaryOperator unaryOperator)
        Either keeps the same offset or updates the purgedOffset if a higher value offset is found in insertReport Snowpipe API
        Parameters:
        unaryOperator - the function to apply on purgedOffset

      • addAndGetFileCountOnStage

        public void addAndGetFileCountOnStage​(long fileCountOnStage)

      • addAndGetFileCountOnIngestion

        public void addAndGetFileCountOnIngestion​(long fileCountOnIngestion)

      • addAndGetFileCountPurged

        public void addAndGetFileCountPurged​(long fileCountPurged)

      • incrementAndGetCleanerRestartCount

        public long incrementAndGetCleanerRestartCount()

      • addAndGetTotalNumberOfRecord

        public void addAndGetTotalNumberOfRecord​(long totalNumberOfRecord)

      • addAndGetTotalSizeOfData

        public void addAndGetTotalSizeOfData​(long totalSizeOfData)

      • addAndGetMemoryUsage

        public void addAndGetMemoryUsage​(long memoryUsage)

      • resetMemoryUsage

        public void resetMemoryUsage()

      • setCleanerRestartCount

        public void setCleanerRestartCount​(long cleanerRestartCount)

      • setAverageKafkaLagMs

        public void setAverageKafkaLagMs​(long averageKafkaLagMs)

      • setAverageKafkaLagRecordCount

        public void setAverageKafkaLagRecordCount​(long averageKafkaLagRecordCount)

      • setAverageIngestionLagMs

        public void setAverageIngestionLagMs​(long averageIngestionLagMs)

      • setAverageIngestionLagFileCount

        public void setAverageIngestionLagFileCount​(long averageIngestionLagFileCount)

      • setAverageCommitLagMs

        public void setAverageCommitLagMs​(long averageCommitLagMs)

      • setAverageCommitLagFileCount

        public void setAverageCommitLagFileCount​(long averageCommitLagFileCount)