Packages

sealed trait WatermarkPropagator extends AnyRef

Interface for propagating watermark. The implementation is not required to be thread-safe, as all methods are expected to be called from the query execution thread. (The guarantee may change on further improvements on Structured Streaming - update implementations if we change the guarantee.)

Linear Supertypes
AnyRef, Any
Known Subclasses
NoOpWatermarkPropagator, PropagateWatermarkSimulator, UseSingleWatermarkPropagator
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. WatermarkPropagator
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long

    Provide the calculated input watermark for eviction for given stateful operator.

  2. abstract def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long

    Provide the calculated input watermark for late events for given stateful operator.

  3. abstract def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit

    Request to propagate watermark among operators based on origin watermark value.

    Request to propagate watermark among operators based on origin watermark value. The result should be input watermark per stateful operator, which Spark will request the value by calling getInputWatermarkXXX with operator ID.

    It is recommended for implementation to cache the result, as Spark can request the propagation multiple times with the same batch ID and origin watermark value.

  4. abstract def purge(batchId: Long): Unit

    Request to clean up cached result on propagation.

    Request to clean up cached result on propagation. Spark will call this method when the given batch ID will be likely to be not re-executed.

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  10. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  12. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  13. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  14. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  15. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  16. def toString(): String
    Definition Classes
    AnyRef → Any
  17. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  18. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  19. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from AnyRef

Inherited from Any

Ungrouped