Packages

  • package root
    Definition Classes
    root
  • package org
    Definition Classes
    root
  • package apache
    Definition Classes
    org
  • package spark
    Definition Classes
    apache
  • package sql

    Allows the execution of relational queries, including those expressed in SQL using Spark.

    Allows the execution of relational queries, including those expressed in SQL using Spark.

    Definition Classes
    spark
  • package execution

    The physical execution component of Spark SQL.

    The physical execution component of Spark SQL. Note that this is a private package. All classes in catalyst are considered an internal API to Spark SQL and are subject to change between minor releases.

    Definition Classes
    sql
  • package streaming
    Definition Classes
    execution
  • case class FlatMapGroupsWithStateExec(func: (Any, Iterator[Any], LogicalGroupState[Any]) => Iterator[Any], keyDeserializer: Expression, valueDeserializer: Expression, initialStateDeserializer: Expression, groupingAttributes: Seq[Attribute], initialStateGroupAttrs: Seq[Attribute], dataAttributes: Seq[Attribute], initialStateDataAttrs: Seq[Attribute], outputObjAttr: Attribute, stateInfo: Option[StatefulOperatorStateInfo], stateEncoder: ExpressionEncoder[Any], stateFormatVersion: Int, outputMode: OutputMode, timeoutConf: GroupStateTimeout, batchTimestampMs: Option[Long], eventTimeWatermark: Option[Long], initialState: SparkPlan, hasInitialState: Boolean, child: SparkPlan) extends SparkPlan with BinaryExecNode with ObjectProducerExec with StateStoreWriter with WatermarkSupport with Product with Serializable

    Physical operator for executing FlatMapGroupsWithState

    Physical operator for executing FlatMapGroupsWithState

    func

    function called on each group

    keyDeserializer

    used to extract the key object for each group.

    valueDeserializer

    used to extract the items in the iterator from an input row.

    initialStateDeserializer

    used to extract the state object from the initialState dataset

    groupingAttributes

    used to group the data

    dataAttributes

    used to read the data

    outputObjAttr

    Defines the output object

    stateEncoder

    used to serialize/deserialize state before calling func

    outputMode

    the output mode of func

    timeoutConf

    used to timeout groups that have not received data in a while

    batchTimestampMs

    processing timestamp of the current batch.

    eventTimeWatermark

    event time watermark for the current batch

    initialState

    the user specified initial state

    hasInitialState

    indicates whether the initial state is provided or not

    child

    the physical plan for the underlying data

    Definition Classes
    streaming
  • InputProcessor

class InputProcessor extends AnyRef

Helper class to update the state store

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. InputProcessor
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new InputProcessor(store: StateStore)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  9. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  10. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  12. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  13. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  14. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  15. def processNewData(dataIter: Iterator[InternalRow]): Iterator[InternalRow]

    For every group, get the key, values and corresponding state and call the function, and return an iterator of rows

  16. def processNewDataWithInitialState(childDataIter: Iterator[InternalRow], initStateIter: Iterator[InternalRow]): Iterator[InternalRow]

    Process the new data iterator along with the initial state.

    Process the new data iterator along with the initial state. The initial state is applied before processing the new data for every key. The user defined function is called only once for every key that has either initial state or data or both.

  17. def processTimedOutState(): Iterator[InternalRow]

    Find the groups that have timeout set and are timing out right now, and call the function

  18. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  19. def toString(): String
    Definition Classes
    AnyRef → Any
  20. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  21. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  22. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()

Inherited from AnyRef

Inherited from Any

Ungrouped