class ContinuousQueuedDataReader extends Closeable
A wrapper for a continuous processing data reader, including a reading queue and epoch markers.
This will be instantiated once per partition - successive calls to compute() in the ContinuousDataSourceRDD will reuse the same reader. This is required to get continuity of offsets across epochs. Each compute() should call the next() method here until null is returned.
- Alphabetic
- By Inheritance
- ContinuousQueuedDataReader
- Closeable
- AutoCloseable
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new ContinuousQueuedDataReader(partitionIndex: Int, reader: ContinuousPartitionReader[InternalRow], schema: StructType, context: TaskContext, dataQueueSize: Int, epochPollIntervalMs: Long)
Type Members
- sealed trait ContinuousRecord extends AnyRef
The record types in the read buffer.
- case class ContinuousRow(row: InternalRow, offset: PartitionOffset) extends ContinuousRecord with Product with Serializable
- class DataReaderThread extends Thread with Logging
The data component of ContinuousQueuedDataReader.
The data component of ContinuousQueuedDataReader. Pushes (row, offset) to the queue when a new row arrives to the ContinuousPartitionReader.
- class EpochMarkerGenerator extends Runnable with Logging
The epoch marker component of ContinuousQueuedDataReader.
The epoch marker component of ContinuousQueuedDataReader. Populates the queue with EpochMarker when a new epoch marker arrives.
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @native()
- def close(): Unit
- Definition Classes
- ContinuousQueuedDataReader → Closeable → AutoCloseable
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable])
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def getPartitionReader(): PartitionReader[InternalRow]
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def next(): InternalRow
Return the next row to be read in the current epoch, or null if the epoch is done.
Return the next row to be read in the current epoch, or null if the epoch is done.
After returning null, the ContinuousDataSourceRDD compute() for the following epoch will call next() again to start getting rows.
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- case object EpochMarker extends ContinuousRecord with Product with Serializable