Packages

c

io.delta.flink.source.internal.enumerator

ContinuousDeltaSourceSplitEnumerator

class ContinuousDeltaSourceSplitEnumerator extends DeltaSourceSplitEnumerator

A SplitEnumerator implementation for org.apache.flink.api.connector.source.Boundedness#CONTINUOUS_UNBOUNDED mode.

This enumerator takes all files that are present in the configured Delta table directory, convert them to DeltaSourceSplit and assigns them to the readers. Once all files from initial snapshot are processed, it starts monitoring Delta table for changes. Each appending data change is converted to DeltaSourceSplit and assigned to readers.

The actual logic for creating the set of DeltaSourceSplit to process, and the logic to decide which reader gets what split can be found DeltaSourceSplitEnumerator and in FileSplitAssigner, respectively.

Linear Supertypes
DeltaSourceSplitEnumerator, SplitEnumerator[DeltaSourceSplit, DeltaEnumeratorStateCheckpoint[DeltaSourceSplit]], CheckpointListener, AutoCloseable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ContinuousDeltaSourceSplitEnumerator
  2. DeltaSourceSplitEnumerator
  3. SplitEnumerator
  4. CheckpointListener
  5. AutoCloseable
  6. AnyRef
  7. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new ContinuousDeltaSourceSplitEnumerator(deltaTablePath: Path, continuousTableProcessor: ContinuousTableProcessor, splitAssigner: FileSplitAssigner, enumContext: SplitEnumeratorContext[DeltaSourceSplit])

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def addReader(subtaskId: Int): Unit
    Definition Classes
    DeltaSourceSplitEnumerator → SplitEnumerator
    Annotations
    @Override()
  5. def addSplits(splits: List[DeltaSourceSplit]): Unit
    Attributes
    protected[enumerator]
    Definition Classes
    DeltaSourceSplitEnumerator
    Annotations
    @SuppressWarnings()
  6. def addSplitsBack(splits: List[DeltaSourceSplit], subtaskId: Int): Unit
    Definition Classes
    DeltaSourceSplitEnumerator → SplitEnumerator
    Annotations
    @Override()
  7. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  8. def assignSplits(): AssignSplitStatus
    Attributes
    protected[enumerator]
    Definition Classes
    DeltaSourceSplitEnumerator
  9. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  10. def close(): Unit
    Definition Classes
    DeltaSourceSplitEnumerator → SplitEnumerator → AutoCloseable
    Annotations
    @Override()
  11. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  12. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  13. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  14. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  15. def getRemainingSplits(): Collection[DeltaSourceSplit]
    Attributes
    protected[enumerator]
    Definition Classes
    DeltaSourceSplitEnumerator
    Annotations
    @SuppressWarnings()
  16. def handleNoMoreSplits(subtaskId: Int): Unit

    The implementation of this method should handle case, where there is no more splits that could be assigned to Source Readers.

    The implementation of this method should handle case, where there is no more splits that could be assigned to Source Readers.

    This method is called by String) method.

    subtaskId

    the subtask id of the source reader who sent the source spit request event.

    Attributes
    protected[enumerator]
    Definition Classes
    ContinuousDeltaSourceSplitEnumerator → DeltaSourceSplitEnumerator
    Annotations
    @Override()
  17. def handleSourceEvent(arg0: Int, arg1: SourceEvent): Unit
    Definition Classes
    SplitEnumerator
  18. def handleSplitRequest(subtaskId: Int, requesterHostname: String): Unit
    Definition Classes
    DeltaSourceSplitEnumerator → SplitEnumerator
    Annotations
    @Override()
  19. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  20. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  21. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  22. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  23. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  24. def notifyCheckpointAborted(arg0: Long): Unit
    Definition Classes
    CheckpointListener
    Annotations
    @throws( classOf[java.lang.Exception] )
  25. def notifyCheckpointComplete(arg0: Long): Unit
    Definition Classes
    SplitEnumerator → CheckpointListener
    Annotations
    @throws( classOf[java.lang.Exception] )
  26. def snapshotState(checkpointId: Long): DeltaEnumeratorStateCheckpoint[DeltaSourceSplit]
    Definition Classes
    ContinuousDeltaSourceSplitEnumerator → SplitEnumerator
    Annotations
    @Override()
  27. def start(): Unit

    Starts Delta table processing.

    Starts Delta table processing.

    Definition Classes
    ContinuousDeltaSourceSplitEnumerator → SplitEnumerator
    Annotations
    @Override()
  28. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  29. def toString(): String
    Definition Classes
    AnyRef → Any
  30. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  31. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  32. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from DeltaSourceSplitEnumerator

Inherited from SplitEnumerator[DeltaSourceSplit, DeltaEnumeratorStateCheckpoint[DeltaSourceSplit]]

Inherited from CheckpointListener

Inherited from AutoCloseable

Inherited from AnyRef

Inherited from Any

Ungrouped