Packages

class StreamingQueryListenerBus extends SparkListener with ListenerBus[StreamingQueryListener, Event]

A bus to forward events to StreamingQueryListeners. This one will send received StreamingQueryListener.Events to the Spark listener bus. It also registers itself with Spark listener bus, so that it can receive StreamingQueryListener.Events and dispatch them to StreamingQueryListeners.

Note 1: Each bus and its registered listeners are associated with a single SparkSession and StreamingQueryManager. So this bus will dispatch events to registered listeners for only those queries that were started in the associated SparkSession.

Note 2: To rebuild Structured Streaming UI in SHS, this bus will be registered into org.apache.spark.scheduler.ReplayListenerBus. We check sparkListenerBus defined or not to determine how to process StreamingQueryListener.Event. If false, it means this bus is used to replay all streaming query event from eventLog.

Linear Supertypes
ListenerBus[StreamingQueryListener, Event], Logging, SparkListener, SparkListenerInterface, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. StreamingQueryListenerBus
  2. ListenerBus
  3. Logging
  4. SparkListener
  5. SparkListenerInterface
  6. AnyRef
  7. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new StreamingQueryListenerBus(sparkListenerBus: Option[LiveListenerBus])

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def addListener(listener: StreamingQueryListener): Unit
    Definition Classes
    ListenerBus
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  7. def doPostEvent(listener: StreamingQueryListener, event: Event): Unit

    Dispatch events to registered StreamingQueryListeners.

    Dispatch events to registered StreamingQueryListeners. Only the events associated queries started in the same SparkSession as this ListenerBus will be dispatched to the listeners.

    Attributes
    protected
    Definition Classes
    StreamingQueryListenerBus → ListenerBus
  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  10. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  11. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. def getTimer(listener: StreamingQueryListener): Option[Timer]
    Attributes
    protected
    Definition Classes
    ListenerBus
  13. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  14. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  15. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  16. def isIgnorableException(e: Throwable): Boolean
    Attributes
    protected
    Definition Classes
    ListenerBus
  17. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  18. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  19. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  20. def logDebug(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  21. def logDebug(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  22. def logError(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  23. def logError(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  24. def logInfo(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  25. def logInfo(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  26. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  27. def logTrace(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  28. def logTrace(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  29. def logWarning(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  30. def logWarning(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  31. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  32. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  33. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  34. def onApplicationEnd(applicationEnd: SparkListenerApplicationEnd): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  35. def onApplicationStart(applicationStart: SparkListenerApplicationStart): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  36. def onBlockManagerAdded(blockManagerAdded: SparkListenerBlockManagerAdded): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  37. def onBlockManagerRemoved(blockManagerRemoved: SparkListenerBlockManagerRemoved): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  38. def onBlockUpdated(blockUpdated: SparkListenerBlockUpdated): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  39. def onEnvironmentUpdate(environmentUpdate: SparkListenerEnvironmentUpdate): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  40. def onExecutorAdded(executorAdded: SparkListenerExecutorAdded): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  41. def onExecutorBlacklisted(executorBlacklisted: SparkListenerExecutorBlacklisted): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  42. def onExecutorBlacklistedForStage(executorBlacklistedForStage: SparkListenerExecutorBlacklistedForStage): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  43. def onExecutorExcluded(executorExcluded: SparkListenerExecutorExcluded): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  44. def onExecutorExcludedForStage(executorExcludedForStage: SparkListenerExecutorExcludedForStage): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  45. def onExecutorMetricsUpdate(executorMetricsUpdate: SparkListenerExecutorMetricsUpdate): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  46. def onExecutorRemoved(executorRemoved: SparkListenerExecutorRemoved): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  47. def onExecutorUnblacklisted(executorUnblacklisted: SparkListenerExecutorUnblacklisted): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  48. def onExecutorUnexcluded(executorUnexcluded: SparkListenerExecutorUnexcluded): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  49. def onJobEnd(jobEnd: SparkListenerJobEnd): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  50. def onJobStart(jobStart: SparkListenerJobStart): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  51. def onNodeBlacklisted(nodeBlacklisted: SparkListenerNodeBlacklisted): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  52. def onNodeBlacklistedForStage(nodeBlacklistedForStage: SparkListenerNodeBlacklistedForStage): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  53. def onNodeExcluded(nodeExcluded: SparkListenerNodeExcluded): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  54. def onNodeExcludedForStage(nodeExcludedForStage: SparkListenerNodeExcludedForStage): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  55. def onNodeUnblacklisted(nodeUnblacklisted: SparkListenerNodeUnblacklisted): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  56. def onNodeUnexcluded(nodeUnexcluded: SparkListenerNodeUnexcluded): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  57. def onOtherEvent(event: SparkListenerEvent): Unit
    Definition Classes
    StreamingQueryListenerBus → SparkListener → SparkListenerInterface
  58. def onResourceProfileAdded(event: SparkListenerResourceProfileAdded): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  59. def onSpeculativeTaskSubmitted(speculativeTask: SparkListenerSpeculativeTaskSubmitted): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  60. def onStageCompleted(stageCompleted: SparkListenerStageCompleted): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  61. def onStageExecutorMetrics(executorMetrics: SparkListenerStageExecutorMetrics): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  62. def onStageSubmitted(stageSubmitted: SparkListenerStageSubmitted): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  63. def onTaskEnd(taskEnd: SparkListenerTaskEnd): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  64. def onTaskGettingResult(taskGettingResult: SparkListenerTaskGettingResult): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  65. def onTaskStart(taskStart: SparkListenerTaskStart): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  66. def onUnpersistRDD(unpersistRDD: SparkListenerUnpersistRDD): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  67. def onUnschedulableTaskSetAdded(unschedulableTaskSetAdded: SparkListenerUnschedulableTaskSetAdded): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  68. def onUnschedulableTaskSetRemoved(unschedulableTaskSetRemoved: SparkListenerUnschedulableTaskSetRemoved): Unit
    Definition Classes
    SparkListener → SparkListenerInterface
  69. def post(event: Event): Unit

    Post a StreamingQueryListener event to the added StreamingQueryListeners.

    Post a StreamingQueryListener event to the added StreamingQueryListeners. Note that only the QueryStarted event is posted to the listener synchronously. Other events are dispatched to Spark listener bus. This method is guaranteed to be called by queries in the same SparkSession as this listener.

  70. def postToAll(event: Event): Unit

    Override the parent postToAll to remove the query id from activeQueryRunIds after all the listeners process QueryTerminatedEvent.

    Override the parent postToAll to remove the query id from activeQueryRunIds after all the listeners process QueryTerminatedEvent. (SPARK-19594)

    Definition Classes
    StreamingQueryListenerBus → ListenerBus
  71. final def removeAllListeners(): Unit
    Definition Classes
    ListenerBus
  72. final def removeListener(listener: StreamingQueryListener): Unit
    Definition Classes
    ListenerBus
  73. def removeListenerOnError(listener: StreamingQueryListener): Unit
    Definition Classes
    ListenerBus
  74. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  75. def toString(): String
    Definition Classes
    AnyRef → Any
  76. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  77. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  78. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()

Inherited from ListenerBus[StreamingQueryListener, Event]

Inherited from Logging

Inherited from SparkListener

Inherited from SparkListenerInterface

Inherited from AnyRef

Inherited from Any

Ungrouped