Package org.apache.beam.runners.spark.metrics.sink

Class CsvSink

  • All Implemented Interfaces:
    org.apache.spark.metrics.sink.Sink
    public class CsvSink
extends java.lang.Object
implements org.apache.spark.metrics.sink.Sink
    A Sink for Spark's metric system reporting metrics (including Beam step metrics) to a CSV file.

    The sink is configured using Spark configuration parameters, for example: 

    
 "spark.metrics.conf.*.sink.csv.class"="org.apache.beam.runners.spark.metrics.sink.CsvSink"
 "spark.metrics.conf.*.sink.csv.directory"="<output_directory>"
 "spark.metrics.conf.*.sink.csv.period"=10
 "spark.metrics.conf.*.sink.csv.unit"=seconds

    • Constructor Summary

      Constructors 
      Constructor Description
      CsvSink​(java.util.Properties properties, com.codahale.metrics.MetricRegistry metricRegistry)
      Constructor for Spark 3.2.x and later.
      CsvSink​(java.util.Properties properties, com.codahale.metrics.MetricRegistry metricRegistry, org.apache.spark.SecurityManager securityMgr)
      Constructor for Spark 3.1.x and earlier.

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void report()  
      void start()  
      void stop()  

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Constructor Detail

      • CsvSink

        public CsvSink​(java.util.Properties properties,
               com.codahale.metrics.MetricRegistry metricRegistry,
               org.apache.spark.SecurityManager securityMgr)
        Constructor for Spark 3.1.x and earlier.

      • CsvSink

        public CsvSink​(java.util.Properties properties,
               com.codahale.metrics.MetricRegistry metricRegistry)
        Constructor for Spark 3.2.x and later.

    • Method Detail

      • start

        public void start()
        Specified by:
        start in interface org.apache.spark.metrics.sink.Sink

      • stop

        public void stop()
        Specified by:
        stop in interface org.apache.spark.metrics.sink.Sink

      • report

        public void report()
        Specified by:
        report in interface org.apache.spark.metrics.sink.Sink