org.apache.spark.sql.execution.streaming.state
StateStoreProvider
Companion object StateStoreProvider
trait StateStoreProvider extends AnyRef
Trait representing a provider that provide StateStore instances representing versions of state data.
The life cycle of a provider and its provide stores are as follows.
- A StateStoreProvider is created in a executor for each unique StateStoreId when the first batch of a streaming query is executed on the executor. All subsequent batches reuse this provider instance until the query is stopped.
- Every batch of streaming data request a specific version of the state data by invoking
getStore(version) which returns an instance of StateStore through which the required
version of the data can be accessed. It is the responsible of the provider to populate
this store with context information like the schema of keys and values, etc.
- After the streaming query is stopped, the created provider instances are lazily disposed off.
- Alphabetic
- By Inheritance
- StateStoreProvider
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Abstract Value Members
- abstract def close(): Unit
Called when the provider instance is unloaded from the executor
- abstract def getStore(version: Long): StateStore
Return an instance of StateStore representing state data of the given version
- abstract def init(stateStoreId: StateStoreId, keySchema: StructType, valueSchema: StructType, numColsPrefixKey: Int, storeConfs: StateStoreConf, hadoopConf: Configuration): Unit
Initialize the provide with more contextual information from the SQL operator.
Initialize the provide with more contextual information from the SQL operator. This method will be called first after creating an instance of the StateStoreProvider by reflection.
- stateStoreId
Id of the versioned StateStores that this provider will generate
- keySchema
Schema of keys to be stored
- valueSchema
Schema of value to be stored
- numColsPrefixKey
The number of leftmost columns to be used as prefix key. A value not greater than 0 means the operator doesn't activate prefix key, and the operator should not call prefixScan method in StateStore.
- storeConfs
Configurations used by the StateStores
- hadoopConf
Hadoop configuration that could be used by StateStore to save state data
- abstract def stateStoreId: StateStoreId
Return the id of the StateStores this provider will generate.
Return the id of the StateStores this provider will generate. Should be the same as the one passed in init().
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @native()
- def doMaintenance(): Unit
Optional method for providers to allow for background maintenance (e.g.
Optional method for providers to allow for background maintenance (e.g. compactions)
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable])
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def getReadStore(version: Long): ReadStateStore
Return an instance of ReadStateStore representing state data of the given version.
Return an instance of ReadStateStore representing state data of the given version. By default it will return the same instance as getStore(version) but wrapped to prevent modification. Providers can override and return optimized version of ReadStateStore based on the fact the instance will be only used for reading.
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- def supportedCustomMetrics: Seq[StateStoreCustomMetric]
Optional custom metrics that the implementation may want to report.
Optional custom metrics that the implementation may want to report.
- Note
The StateStore objects created by this provider must report the same custom metrics (specifically, same names) through
StateStore.metrics.
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()