Package org.apache.beam.runners.spark.translation

Internal translators for running Beam pipelines on Spark.

Interface Summary

Interface	Description
Dataset	Holder for Spark RDD/DStream.
SparkCombineFn.WindowedAccumulator<InputT,ValueT,AccumT,ImplT extends SparkCombineFn.WindowedAccumulator<InputT,ValueT,AccumT,ImplT>>	Accumulator of WindowedValues holding values for different windows.
SparkPipelineTranslator	Translator to support translation between Beam transformations and Spark transformations.
SparkPortablePipelineTranslator<T extends SparkTranslationContext>	Interface for portable Spark translators.
TransformEvaluator<TransformT extends org.apache.beam.sdk.transforms.PTransform<?,?>>	Describe a PTransform evaluator.

Class Summary

Class	Description
BoundedDataset<T>	Holds an RDD or values for deferred conversion to an RDD if needed.
EvaluationContext	The EvaluationContext allows us to define pipeline instructions and translate between PObject<T>s or PCollection<T>s and Ts or DStreams/RDDs of Ts.
GroupCombineFunctions	A set of group/combine functions to apply to Spark RDDs.
GroupNonMergingWindowsFunctions	Functions for GroupByKey with Non-Merging windows translations to Spark.
MultiDoFnFunction<InputT,OutputT>	DoFunctions ignore outputs that are not the main output.
ReifyTimestampsAndWindowsFunction<K,V>	Simple Function to bring the windowing information into the value from the implicit background representation of the PCollection.
SparkAssignWindowFn<T,W extends org.apache.beam.sdk.transforms.windowing.BoundedWindow>	An implementation of Window.Assign for the Spark runner.
SparkBatchPortablePipelineTranslator	Translates a bounded portable pipeline into a Spark job.
SparkBatchPortablePipelineTranslator.IsSparkNativeTransform	Predicate to determine whether a URN is a Spark native transform.
SparkCombineFn<InputT,ValueT,AccumT,OutputT>	A CombineFnBase.GlobalCombineFn with a CombineWithContext.Context for the SparkRunner.
SparkContextFactory
SparkExecutableStageContextFactory	Singleton class that contains one ExecutableStageContext.Factory per job.
SparkPCollectionView	SparkPCollectionView is used to pass serialized views to lambdas.
SparkStreamingPortablePipelineTranslator	Translates an unbounded portable pipeline into a Spark job.
SparkStreamingTranslationContext	Translation context used to lazily store Spark datasets during streaming portable pipeline translation and compute them after translation.
SparkTranslationContext	Translation context used to lazily store Spark data sets during portable pipeline translation and compute them after translation.
TransformTranslator	Supports translation between a Beam transform, and Spark's operations on RDDs.
TransformTranslator.Translator	Translator matches Beam transformation with the appropriate evaluator.
TranslationUtils	A set of utilities to help translating Beam transformations into Spark transformations.
TranslationUtils.CombineGroupedValues<K,InputT,OutputT>	A SparkCombineFn function applied to grouped KVs.
TranslationUtils.TupleTagFilter<V>	A utility class to filter TupleTags.
ValueAndCoderKryoSerializer<T>	Kryo serializer for ValueAndCoderLazySerializable.
ValueAndCoderLazySerializable<T>	A holder object that lets you serialize an element with a Coder with minimal wasted space.

Enum Summary

Enum	Description
SparkCombineFn.WindowedAccumulator.Type	Type of the accumulator.