Class EvaluationContext
- java.lang.Object
-
- org.apache.beam.runners.spark.structuredstreaming.translation.EvaluationContext
-
@Internal public final class EvaluationContext extends java.lang.ObjectTheEvaluationContextis the result of a pipelinetranslationand can be used to evaluate / run the pipeline.However, in some cases pipeline translation involves the early evaluation of some parts of the pipeline. For example, this is necessary to materialize side-inputs. The
EvaluationContextwon't re-evaluate such datasets.
-
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description static <T extends @NonNull java.lang.Object>
T[]collect(java.lang.String name, org.apache.spark.sql.Dataset<T> ds)The purpose of this utility is to mark the evaluation of Spark actions, both during Pipeline translation, when evaluation is required, and when finally evaluating the pipeline.voidevaluate()Trigger evaluation of all leaf datasets.static <T> voidevaluate(java.lang.String name, org.apache.spark.sql.Dataset<T> ds)The purpose of this utility is to mark the evaluation of Spark actions, both during Pipeline translation, when evaluation is required, and when finally evaluating the pipeline.org.apache.spark.sql.SparkSessiongetSparkSession()
-
-
-
Method Detail
-
evaluate
public void evaluate()
Trigger evaluation of all leaf datasets.
-
evaluate
public static <T> void evaluate(java.lang.String name, org.apache.spark.sql.Dataset<T> ds)The purpose of this utility is to mark the evaluation of Spark actions, both during Pipeline translation, when evaluation is required, and when finally evaluating the pipeline.
-
collect
public static <T extends @NonNull java.lang.Object> T[] collect(java.lang.String name, org.apache.spark.sql.Dataset<T> ds)The purpose of this utility is to mark the evaluation of Spark actions, both during Pipeline translation, when evaluation is required, and when finally evaluating the pipeline.
-
getSparkSession
public org.apache.spark.sql.SparkSession getSparkSession()
-
-