class UseSingleWatermarkPropagator extends WatermarkPropagator
This implementation uses a single global watermark for late events and eviction.
This implementation provides the behavior before Structured Streaming supports multiple stateful operators. (prior to SPARK-40925) This is only used for compatibility mode.
- Alphabetic
- By Inheritance
- UseSingleWatermarkPropagator
- WatermarkPropagator
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
- new UseSingleWatermarkPropagator()
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
Provide the calculated input watermark for eviction for given stateful operator.
Provide the calculated input watermark for eviction for given stateful operator.
- Definition Classes
- UseSingleWatermarkPropagator → WatermarkPropagator
-
def
getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
Provide the calculated input watermark for late events for given stateful operator.
Provide the calculated input watermark for late events for given stateful operator.
- Definition Classes
- UseSingleWatermarkPropagator → WatermarkPropagator
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
def
propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
Request to propagate watermark among operators based on origin watermark value.
Request to propagate watermark among operators based on origin watermark value. The result should be input watermark per stateful operator, which Spark will request the value by calling getInputWatermarkXXX with operator ID.
It is recommended for implementation to cache the result, as Spark can request the propagation multiple times with the same batch ID and origin watermark value.
- Definition Classes
- UseSingleWatermarkPropagator → WatermarkPropagator
-
def
purge(batchId: Long): Unit
Request to clean up cached result on propagation.
Request to clean up cached result on propagation. Spark will call this method when the given batch ID will be likely to be not re-executed.
- Definition Classes
- UseSingleWatermarkPropagator → WatermarkPropagator
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()