Package org.apache.beam.runners.spark.translation

Interface SparkPortablePipelineTranslator<T extends SparkTranslationContext>

All Known Implementing Classes:

SparkBatchPortablePipelineTranslator, SparkStreamingPortablePipelineTranslator

public interface SparkPortablePipelineTranslator<T extends SparkTranslationContext>

Interface for portable Spark translators. This allows for a uniform invocation pattern for pipeline translation between streaming and batch runners.

Method Summary

Modifier and Type	Method	Description
T	createTranslationContext(org.apache.spark.api.java.JavaSparkContext jsc, SparkPipelineOptions options, org.apache.beam.runners.fnexecution.provisioning.JobInfo jobInfo)
java.util.Set<java.lang.String>	knownUrns()
void	translate(org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline pipeline, T context)	Translates the given pipeline.

Method Detail

knownUrns

java.util.Set<java.lang.String> knownUrns()

translate

void translate(org.apache.beam.model.pipeline.v1.RunnerApi.Pipeline pipeline,
               T context)

Translates the given pipeline.

createTranslationContext

T createTranslationContext(org.apache.spark.api.java.JavaSparkContext jsc,
                           SparkPipelineOptions options,
                           org.apache.beam.runners.fnexecution.provisioning.JobInfo jobInfo)