Package org.apache.beam.runners.spark.structuredstreaming.translation

Class TransformTranslator<InT extends org.apache.beam.sdk.values.PInput,​OutT extends org.apache.beam.sdk.values.POutput,​TransformT extends org.apache.beam.sdk.transforms.PTransform<InT,​OutT>>

  • @Internal
public abstract class TransformTranslator<InT extends org.apache.beam.sdk.values.PInput,​OutT extends org.apache.beam.sdk.values.POutput,​TransformT extends org.apache.beam.sdk.transforms.PTransform<InT,​OutT>>
extends java.lang.Object
    A TransformTranslator provides the capability to translate a specific primitive or composite PTransform into its Spark correspondence.

    WARNING: TransformTranslators should never be serializable! This could easily hide situations where unnecessary references leak into Spark closures.

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description
      protected class  TransformTranslator.Context
      Available mutable context to translate a PTransform.

    • Field Summary

      Fields 
      Modifier and Type Field Description
      protected float complexityFactor  

    • Constructor Summary

      Constructors 
      Modifier Constructor Description
      protected TransformTranslator​(float complexityFactor)  

    • Method Summary

      All Methods Instance Methods Abstract Methods Concrete Methods 
      Modifier and Type Method Description
      protected boolean canTranslate​(TransformT transform)
      Checks if a composite / primitive transform can be translated.
      protected abstract void translate​(TransformT transform, TransformTranslator.Context cxt)  
      protected <T> org.apache.beam.sdk.coders.Coder<org.apache.beam.sdk.transforms.windowing.BoundedWindow> windowCoder​(org.apache.beam.sdk.values.PCollection<T> pc)  

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Field Detail

      • complexityFactor

        protected final float complexityFactor

    • Constructor Detail

      • TransformTranslator

        protected TransformTranslator​(float complexityFactor)

    • Method Detail

      • canTranslate

        protected boolean canTranslate​(TransformT transform)
        Checks if a composite / primitive transform can be translated. Composites that cannot be translated as is, will be exploded further for translation of their parts.

        This returns true by default and should be overridden where necessary.

        Throws:
        java.lang.RuntimeException - If a transform uses unsupported features, an exception shall be thrown to give early feedback before any part of the pipeline is run.

      • windowCoder

        protected <T> org.apache.beam.sdk.coders.Coder<org.apache.beam.sdk.transforms.windowing.BoundedWindow> windowCoder​(org.apache.beam.sdk.values.PCollection<T> pc)