Package org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions

Class SparkSideInputReader

  • All Implemented Interfaces:
    java.io.Serializable, org.apache.beam.runners.core.SideInputReader
    public class SparkSideInputReader
extends java.lang.Object
implements org.apache.beam.runners.core.SideInputReader, java.io.Serializable
    SideInputReader using broadcasted SideInputValues.
    See Also:
    Serialized Form

    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      <T> boolean contains​(org.apache.beam.sdk.values.PCollectionView<T> view)  
      static org.apache.beam.runners.core.SideInputReader create​(java.util.Map<java.lang.String,​org.apache.spark.broadcast.Broadcast<SideInputValues<?>>> sideInputs)
      Creates a SideInputReader for Spark from a map of PCollectionView tag ids and the corresponding broadcasted SideInputValues.
      static org.apache.beam.runners.core.SideInputReader empty()  
      <T> @Nullable T get​(org.apache.beam.sdk.values.PCollectionView<T> view, org.apache.beam.sdk.transforms.windowing.BoundedWindow window)  
      boolean isEmpty()  
      static void validateMaterializations​(java.lang.Iterable<org.apache.beam.sdk.values.PCollectionView<?>> views)  

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Method Detail

      • empty

        public static org.apache.beam.runners.core.SideInputReader empty()

      • create

        public static org.apache.beam.runners.core.SideInputReader create​(java.util.Map<java.lang.String,​org.apache.spark.broadcast.Broadcast<SideInputValues<?>>> sideInputs)
        Creates a SideInputReader for Spark from a map of PCollectionView tag ids and the corresponding broadcasted SideInputValues.

        Note, the materialization of respective PCollectionViews should be validated ahead of time before any costly creation and broadcast of SideInputValues.

      • validateMaterializations

        public static void validateMaterializations​(java.lang.Iterable<org.apache.beam.sdk.values.PCollectionView<?>> views)

      • get

        public <T> @Nullable T get​(org.apache.beam.sdk.values.PCollectionView<T> view,
                           org.apache.beam.sdk.transforms.windowing.BoundedWindow window)
        Specified by:
        get in interface org.apache.beam.runners.core.SideInputReader

      • contains

        public <T> boolean contains​(org.apache.beam.sdk.values.PCollectionView<T> view)
        Specified by:
        contains in interface org.apache.beam.runners.core.SideInputReader

      • isEmpty

        public boolean isEmpty()
        Specified by:
        isEmpty in interface org.apache.beam.runners.core.SideInputReader