Package org.apache.beam.runners.spark

Class SparkRunnerDebugger

  • public final class SparkRunnerDebugger
extends org.apache.beam.sdk.PipelineRunner<SparkPipelineResult>
    Pipeline runner which translates a Beam pipeline into equivalent Spark operations, without running them. Used for debugging purposes.

    Example: 

    
 SparkPipelineOptions options = PipelineOptionsFactory.as(SparkPipelineOptions.class);
 options.setRunner(SparkRunnerDebugger.class);
 Pipeline pipeline = Pipeline.create(options);
 SparkRunnerDebugger.DebugSparkPipelineResult result =
     (SparkRunnerDebugger.DebugSparkPipelineResult) pipeline.run();
 String sparkPipeline = result.getDebugString();

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description
      static class  SparkRunnerDebugger.DebugSparkPipelineResult
      PipelineResult of running a Pipeline using SparkRunnerDebugger Use SparkRunnerDebugger.DebugSparkPipelineResult.getDebugString() to get a String representation of the Pipeline translated into Spark native operations.

    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      static SparkRunnerDebugger fromOptions​(org.apache.beam.sdk.options.PipelineOptions options)  
      SparkPipelineResult run​(org.apache.beam.sdk.Pipeline pipeline)  

      • Methods inherited from class org.apache.beam.sdk.PipelineRunner

        create, run, run

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait