Packages

trait ProgressReporter extends Logging

Responsible for continually reporting statistics about the amount of data processed as well as latency for a streaming query. This trait is designed to be mixed into the StreamExecution, who is responsible for calling startTrigger and finishTrigger at the appropriate times. Additionally, the status can updated with updateStatusMessage to allow reporting on the streams current state (i.e. "Fetching more data").

Linear Supertypes
Logging, AnyRef, Any
Known Subclasses
AsyncProgressTrackingMicroBatchExecution, MicroBatchExecution, StreamExecution, ContinuousExecution
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ProgressReporter
  2. Logging
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Type Members

  1. case class ExecutionStats(inputRows: Map[SparkDataStream, Long], stateOperators: Seq[StateOperatorProgress], eventTimeStats: Map[String, String]) extends Product with Serializable

Abstract Value Members

  1. abstract def currentBatchId: Long
    Attributes
    protected
  2. abstract def id: UUID
    Attributes
    protected
  3. abstract def lastExecution: QueryExecution
    Attributes
    protected
  4. abstract def logicalPlan: LogicalPlan
    Attributes
    protected
  5. abstract def name: String
    Attributes
    protected
  6. abstract def newData: Map[SparkDataStream, LogicalPlan]
    Attributes
    protected
  7. abstract def offsetSeqMetadata: OffsetSeqMetadata
    Attributes
    protected
  8. abstract def postEvent(event: Event): Unit
    Attributes
    protected
  9. abstract def runId: UUID
    Attributes
    protected
  10. abstract def sink: Table
    Attributes
    protected
  11. abstract def sinkCommitProgress: Option[StreamWriterCommitProgress]
    Attributes
    protected
  12. abstract def sources: Seq[SparkDataStream]
    Attributes
    protected
  13. abstract def sparkSession: SparkSession
    Attributes
    protected
  14. abstract def triggerClock: Clock
    Attributes
    protected

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  6. val currentStatus: StreamingQueryStatus
    Attributes
    protected
    Annotations
    @volatile()
  7. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  8. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  9. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  10. def finishTrigger(hasNewData: Boolean, hasExecuted: Boolean): Unit

    Finalizes the query progress and adds it to list of recent status updates.

    Finalizes the query progress and adds it to list of recent status updates.

    hasNewData

    Whether the sources of this stream had new data for this trigger.

    hasExecuted

    Whether any batch was executed during this trigger. Streaming queries that perform stateful aggregations with timeouts can still run batches even though the sources don't have any new data.

    Attributes
    protected
  11. def formatTimestamp(millis: Long): String
    Attributes
    protected
  12. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  13. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  14. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  15. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  16. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  17. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  18. def lastProgress: StreamingQueryProgress

    Returns the most recent query progress update or null if there were no progress updates.

  19. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  20. def logDebug(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  21. def logDebug(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  22. def logError(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  23. def logError(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  24. def logInfo(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  25. def logInfo(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  26. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  27. def logTrace(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  28. def logTrace(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  29. def logWarning(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  30. def logWarning(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  31. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  32. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  33. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  34. def recentProgress: Array[StreamingQueryProgress]

    Returns an array containing the most recent query progress updates.

  35. def recordTriggerOffsets(from: StreamProgress, to: StreamProgress, latest: StreamProgress): Unit

    Record the offsets range this trigger will process.

    Record the offsets range this trigger will process. Call this before updating committedOffsets in StreamExecution to make sure that the correct range is recorded.

    Attributes
    protected
  36. def reportTimeTaken[T](triggerDetailKey: String)(body: => T): T

    Records the duration of running body for the next query progress update.

    Records the duration of running body for the next query progress update.

    Attributes
    protected
  37. def startTrigger(): Unit

    Begins recording statistics about query progress for a given trigger.

    Begins recording statistics about query progress for a given trigger.

    Attributes
    protected
  38. def status: StreamingQueryStatus

    Returns the current status of the query.

  39. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  40. def toString(): String
    Definition Classes
    AnyRef → Any
  41. def updateStatusMessage(message: String): Unit

    Updates the message returned in status.

    Updates the message returned in status.

    Attributes
    protected
  42. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  43. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  44. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped