Class MetricsUtil


  • public class MetricsUtil
    extends Object
    All metrics related constants. Mainly for JMX
    • Field Detail

      • FILE_COUNT_ON_INGESTION

        public static final String FILE_COUNT_ON_INGESTION
        Number of files we call insertFiles API in snowpipe. Note: There is currently a limitation of 5k files being sent to a single rest request. So these metric has no one to one relation between files and number of REST API calls. Number of REST API call for insertFiles can be larger than this value. The value drop backs to zero if there are no more files to be ingested.
        See Also:
        Constant Field Values
      • FILE_COUNT_ON_STAGE

        public static final String FILE_COUNT_ON_STAGE
        Number of files currently on an internal stage. The value will be decremented once files are being purged. These gives an estimate on how many files are present on an internal stage at any given point of time. (The value is decremented if purged few files due to ingestion status)
        See Also:
        Constant Field Values
      • FILE_COUNT_PURGED

        public static final String FILE_COUNT_PURGED
        Number of files purged from internal stage because we were able to verify the ingestion status
        See Also:
        Constant Field Values
      • FILE_COUNT_TABLE_STAGE_BROKEN_RECORD

        public static final String FILE_COUNT_TABLE_STAGE_BROKEN_RECORD
        Number of files present on table stage because files corresponds to broken offset (Broken record)
        See Also:
        Constant Field Values
      • FILE_COUNT_TABLE_STAGE_INGESTION_FAIL

        public static final String FILE_COUNT_TABLE_STAGE_INGESTION_FAIL
        Number of files present on table stage due to failed ingestion (Missing Ingestion Status).
        See Also:
        Constant Field Values
      • PROCESSED_OFFSET

        public static final String PROCESSED_OFFSET
        Offset number that is most recent inside the buffer (In memory buffer)

        This is updated every time an offset is sent as put API of SinkTask SinkTask.put(Collection)

        See Also:
        Constant Field Values
      • FLUSHED_OFFSET

        public static final String FLUSHED_OFFSET
        Offset number(Record) that is being flushed into an internal stage after the buffer threshold was reached. Buffer can reach its threshold by either time, number of records or size.
        See Also:
        Constant Field Values
      • COMMITTED_OFFSET

        public static final String COMMITTED_OFFSET
        Offset number (Record) for which precommit SinkTask.preCommit(Map) API was called and we called snowpipe's insertFiles API.
        See Also:
        Constant Field Values
      • PURGED_OFFSET

        public static final String PURGED_OFFSET
        Offsets which are being purged from internal stage. (This number is the highest recent most offset which was purged from internal stage)
        See Also:
        Constant Field Values
    • Constructor Detail

      • MetricsUtil

        public MetricsUtil()
    • Method Detail

      • constructMetricName

        public static String constructMetricName​(String partitionName,
                                                 String subDomain,
                                                 String metricName)
        Construct the actual metrics name that will be passed in by dropwizard framework to MetricsJmxReporter.getObjectName(String, String, String) We will prefix actual metric name with partitionName and subcategory of the metric.

        Will be of form partitionName/subDomain/metricName

        Parameters:
        partitionName - partitionNAme based on partition number (pipeName for Snowpipe or partitionChannelKey for Streaming)
        subDomain - categorize this metric (Actual ObjectName creation Logic will be handled in getObjectName)
        metricName - actual Metric name for which we will use Gauge, Meter, Histogram
        Returns:
        concatenized String