Packages

c

ai.chronon.flink.window

BufferedProcessingTimeTrigger

class BufferedProcessingTimeTrigger extends Trigger[Map[String, Any], TimeWindow]

BufferedProcessingTimeTrigger is a custom Trigger that fires at most every 'bufferSizeMillis' within a window. It is intended for incremental window aggregations using event-time semantics.

Purpose: This trigger exists as an optimization to reduce the number of writes to our online store and better handle contention that arises from having hot keys.

Details:

  • The buffer timers are NOT aligned with the UNIX Epoch, they can fire at any timestamp. e.g., if the first event arrives at 14ms, and the buffer size is 100ms, the timer will fire at 114ms.
  • Buffer timers are only scheduled when events come in. If there's a gap in events, this trigger won't fire.

Edge cases handled: - If the (event-time) window closes before the last (processing-time) buffer fires, this trigger will fire the remaining buffered elements before closing.

Example: Window size = 300,000 ms (5 minutes) BufferSizeMillis = 100 ms. Assume we are using this trigger on a GroupBy that counts the number unique IDs see. For simplicity, assume event time and processing time are synchronized (although in practice this is never true)

Event 1: ts = 14 ms, ID = A. preAggregate (a Set that keeps track of all unique IDs seen) = [A] this causes a timer to be set for timestamp = 114 ms. Event 2: ts = 38 ms, ID = B. preAggregate = [A, B] Event 3: ts = 77 ms, ID = B. preAggregate = [A, B] Timer set for 114ms fires. we emit the preAggregate [A, B]. Event 4: ts = 400ms, ID = C. preAggregate = [A,B,C] (we don't purge the previous events when the time fires!) this causes a timer to be set for timestamp = 500 ms Timer set for 500ms fires. we emit the preAggregate [A, B, C].

Linear Supertypes
Trigger[Map[String, Any], TimeWindow], Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. BufferedProcessingTimeTrigger
  2. Trigger
  3. Serializable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new BufferedProcessingTimeTrigger(bufferSizeMillis: Long)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def canMerge(): Boolean
    Definition Classes
    Trigger
  6. def clear(window: TimeWindow, ctx: TriggerContext): Unit

    When a window is being purged (e.g., because it has expired), we delete timers and state.

    When a window is being purged (e.g., because it has expired), we delete timers and state.

    This function is called immediately after our 'onEventTime' which fires at the end of the window. See 'onEventTime' in Flink's 'WindowOperator.java'.

    Definition Classes
    BufferedProcessingTimeTrigger → Trigger
  7. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  10. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  11. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  13. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  14. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  15. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  16. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  17. def onElement(element: Map[String, Any], timestamp: Long, window: TimeWindow, ctx: TriggerContext): TriggerResult

    When an element arrives, set up a processing time trigger to fire after bufferSizeMillis.

    When an element arrives, set up a processing time trigger to fire after bufferSizeMillis. If a timer is already set, we don't want to create a new one.

    Late events are treated the same way as regular events; they will still get buffered.

    Definition Classes
    BufferedProcessingTimeTrigger → Trigger
  18. def onEventTime(timestamp: Long, window: TimeWindow, ctx: TriggerContext): TriggerResult

    Fire any elements left in the buffer if the window ends before the last processing-time timer is fired.

    Fire any elements left in the buffer if the window ends before the last processing-time timer is fired. This can happen because we are using event-time semantics for the window, and processing-time for the buffer timer.

    Flink automatically sets up an event timer for the end of the window (+ allowed lateness) as soon as it sees the first element in it. See 'registerCleanupTimer' in Flink's 'WindowOperator.java'.

    Definition Classes
    BufferedProcessingTimeTrigger → Trigger
  19. def onMerge(arg0: TimeWindow, arg1: OnMergeContext): Unit
    Definition Classes
    Trigger
    Annotations
    @throws( classOf[java.lang.Exception] )
  20. def onProcessingTime(timestamp: Long, window: TimeWindow, ctx: TriggerContext): TriggerResult

    When the processing-time timer set up in onElement fires, we emit the results without purging the window.

    When the processing-time timer set up in onElement fires, we emit the results without purging the window. i.e., we keep the current pre-aggregates/IRs in the window so we can continue aggregating.

    Note: We don't need to PURGE the window anywhere. Flink will do that automatically when a window expires. Flink Docs: "[...] Flink keeps the state of windows until their allowed lateness expires. Once this happens, Flink removes the window and deletes its state [...]".

    Note: In case the app crashes after a processing-time timer is set, but before it fires, it will fire immediately after recovery.

    Definition Classes
    BufferedProcessingTimeTrigger → Trigger
  21. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  22. def toString(): String
    Definition Classes
    AnyRef → Any
  23. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  24. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  25. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from Trigger[Map[String, Any], TimeWindow]

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped