Class MetricsUtil
- java.lang.Object
-
- com.snowflake.kafka.connector.internal.metrics.MetricsUtil
-
public class MetricsUtil extends Object
All metrics related constants. Mainly for JMX
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static classMetricsUtil.EventType
-
Field Summary
Fields Modifier and Type Field Description static StringBUFFER_RECORD_COUNTstatic StringBUFFER_SIZE_BYTESstatic StringBUFFER_SUB_DOMAINstatic StringCOMMITTED_OFFSETOffset number (Record) for which precommitSinkTask.preCommit(Map)API was called and we called snowpipe's insertFiles API.static StringFILE_COUNT_ON_INGESTIONNumber of files we call insertFiles API in snowpipe.static StringFILE_COUNT_ON_STAGENumber of files currently on an internal stage.static StringFILE_COUNT_PURGEDNumber of files purged from internal stage because we were able to verify the ingestion statusstatic StringFILE_COUNT_SUB_DOMAINstatic StringFILE_COUNT_TABLE_STAGE_BROKEN_RECORDNumber of files present on table stage because files corresponds to broken offset (Broken record)static StringFILE_COUNT_TABLE_STAGE_INGESTION_FAILNumber of files present on table stage due to failed ingestion (Missing Ingestion Status).static StringFLUSHED_OFFSETOffset number(Record) that is being flushed into an internal stage after the buffer threshold was reached.static StringJMX_METRIC_PREFIXstatic StringLATENCY_SUB_DOMAINstatic StringLATEST_CONSUMER_OFFSETstatic StringOFFSET_PERSISTED_IN_SNOWFLAKESeeTopicPartitionChannelfor offset descriptionstatic StringOFFSET_SUB_DOMAINstatic StringPROCESSED_OFFSETOffset number that is most recent inside the buffer (In memory buffer)static StringPURGED_OFFSETOffsets which are being purged from internal stage.
-
Constructor Summary
Constructors Constructor Description MetricsUtil()
-
Method Summary
All Methods Static Methods Concrete Methods Modifier and Type Method Description static StringconstructMetricName(String partitionName, String subDomain, String metricName)Construct the actual metrics name that will be passed in by dropwizard framework toMetricsJmxReporter.getObjectName(String, String, String)We will prefix actual metric name with partitionName and subcategory of the metric.
-
-
-
Field Detail
-
JMX_METRIC_PREFIX
public static final String JMX_METRIC_PREFIX
- See Also:
- Constant Field Values
-
FILE_COUNT_SUB_DOMAIN
public static final String FILE_COUNT_SUB_DOMAIN
- See Also:
- Constant Field Values
-
FILE_COUNT_ON_INGESTION
public static final String FILE_COUNT_ON_INGESTION
Number of files we call insertFiles API in snowpipe. Note: There is currently a limitation of 5k files being sent to a single rest request. So these metric has no one to one relation between files and number of REST API calls. Number of REST API call for insertFiles can be larger than this value. The value drop backs to zero if there are no more files to be ingested.- See Also:
- Constant Field Values
-
FILE_COUNT_ON_STAGE
public static final String FILE_COUNT_ON_STAGE
Number of files currently on an internal stage. The value will be decremented once files are being purged. These gives an estimate on how many files are present on an internal stage at any given point of time. (The value is decremented if purged few files due to ingestion status)- See Also:
- Constant Field Values
-
FILE_COUNT_PURGED
public static final String FILE_COUNT_PURGED
Number of files purged from internal stage because we were able to verify the ingestion status- See Also:
- Constant Field Values
-
FILE_COUNT_TABLE_STAGE_BROKEN_RECORD
public static final String FILE_COUNT_TABLE_STAGE_BROKEN_RECORD
Number of files present on table stage because files corresponds to broken offset (Broken record)- See Also:
- Constant Field Values
-
FILE_COUNT_TABLE_STAGE_INGESTION_FAIL
public static final String FILE_COUNT_TABLE_STAGE_INGESTION_FAIL
Number of files present on table stage due to failed ingestion (Missing Ingestion Status).- See Also:
- Constant Field Values
-
OFFSET_SUB_DOMAIN
public static final String OFFSET_SUB_DOMAIN
- See Also:
- Constant Field Values
-
PROCESSED_OFFSET
public static final String PROCESSED_OFFSET
Offset number that is most recent inside the buffer (In memory buffer)This is updated every time an offset is sent as put API of SinkTask
SinkTask.put(Collection)- See Also:
- Constant Field Values
-
FLUSHED_OFFSET
public static final String FLUSHED_OFFSET
Offset number(Record) that is being flushed into an internal stage after the buffer threshold was reached. Buffer can reach its threshold by either time, number of records or size.- See Also:
- Constant Field Values
-
COMMITTED_OFFSET
public static final String COMMITTED_OFFSET
Offset number (Record) for which precommitSinkTask.preCommit(Map)API was called and we called snowpipe's insertFiles API.- See Also:
- Constant Field Values
-
PURGED_OFFSET
public static final String PURGED_OFFSET
Offsets which are being purged from internal stage. (This number is the highest recent most offset which was purged from internal stage)- See Also:
- Constant Field Values
-
BUFFER_SUB_DOMAIN
public static final String BUFFER_SUB_DOMAIN
- See Also:
- Constant Field Values
-
BUFFER_SIZE_BYTES
public static final String BUFFER_SIZE_BYTES
- See Also:
- Constant Field Values
-
BUFFER_RECORD_COUNT
public static final String BUFFER_RECORD_COUNT
- See Also:
- Constant Field Values
-
LATENCY_SUB_DOMAIN
public static final String LATENCY_SUB_DOMAIN
- See Also:
- Constant Field Values
-
OFFSET_PERSISTED_IN_SNOWFLAKE
public static final String OFFSET_PERSISTED_IN_SNOWFLAKE
SeeTopicPartitionChannelfor offset description- See Also:
- Constant Field Values
-
LATEST_CONSUMER_OFFSET
public static final String LATEST_CONSUMER_OFFSET
- See Also:
- Constant Field Values
-
-
Method Detail
-
constructMetricName
public static String constructMetricName(String partitionName, String subDomain, String metricName)
Construct the actual metrics name that will be passed in by dropwizard framework toMetricsJmxReporter.getObjectName(String, String, String)We will prefix actual metric name with partitionName and subcategory of the metric.Will be of form partitionName/subDomain/metricName
- Parameters:
partitionName- partitionNAme based on partition number (pipeName for Snowpipe or partitionChannelKey for Streaming)subDomain- categorize this metric (Actual ObjectName creation Logic will be handled in getObjectName)metricName- actual Metric name for which we will use Gauge, Meter, Histogram- Returns:
- concatenized String
-
-