Packages

class ChangesProcessor extends TableProcessorBase with ContinuousTableProcessor

This implementation of TableProcessor process only Delta table changes starting from specified io.delta.standalone.Snapshot version. This implementation does not read Snapshot content.

The Snapshot version is specified by TableMonitor used when creating an instance of ChangesProcessor.

Linear Supertypes
ContinuousTableProcessor, TableProcessorBase, TableProcessor, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ChangesProcessor
  2. ContinuousTableProcessor
  3. TableProcessorBase
  4. TableProcessor
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new ChangesProcessor(deltaTablePath: Path, tableMonitor: TableMonitor, enumContext: SplitEnumeratorContext[DeltaSourceSplit], fileEnumerator: AddFileEnumerator[DeltaSourceSplit], sourceConfiguration: DeltaConnectorConfiguration)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  10. def getSnapshotVersion(): Long

    returns

    A Snapshot version that this processor used as a starting version to get changes from Delta table. The method can return different values for every method call.

    Definition Classes
    ChangesProcessor → TableProcessor
    Annotations
    @Override()
  11. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  13. def isMonitoringForChanges(): Boolean

    returns

    return always true indicating that this processor process only changes.

    Definition Classes
    ChangesProcessor → ContinuousTableProcessor
    Annotations
    @Override()
  14. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  15. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  16. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  17. def prepareSplits(changes: ChangesPerVersion[AddFile], splitFilter: SplitFilter[Path]): List[DeltaSourceSplit]
    Attributes
    protected[processor]
    Definition Classes
    TableProcessorBase
  18. def process(processCallback: Consumer[List[DeltaSourceSplit]]): Unit

    Starts processing changes that were added to Delta table starting from version specified by #currentSnapshotVersion field by converting them to DeltaSourceSplit objects.

    Starts processing changes that were added to Delta table starting from version specified by #currentSnapshotVersion field by converting them to DeltaSourceSplit objects.

    processCallback

    A Consumer callback that will be called after processing all io.delta.standalone.actions.Action and converting them to DeltaSourceSplit. This callback will be executed for every new discovered Delta table version.

    Definition Classes
    ChangesProcessor → TableProcessor
    Annotations
    @Override()
  19. def setUpEnumeratorContext(addFiles: List[AddFile], snapshotVersion: Long): AddFileEnumeratorContext
    Attributes
    protected[processor]
    Definition Classes
    TableProcessorBase
  20. def snapshotState(checkpointBuilder: DeltaEnumeratorStateCheckpointBuilder[DeltaSourceSplit]): DeltaEnumeratorStateCheckpointBuilder[DeltaSourceSplit]

    Add TableProcessor state information to DeltaEnumeratorStateCheckpointBuilder to be stored in Flink's checkpoint.

    Add TableProcessor state information to DeltaEnumeratorStateCheckpointBuilder to be stored in Flink's checkpoint.

    The implementation of this method should add the latest state information to DeltaEnumeratorStateCheckpointBuilder needed to recreate TableProcessor instance during Flink recovery.

    checkpointBuilder

    the DeltaEnumeratorStateCheckpointBuilder instance that should be updated with TableProcessor state information.

    returns

    the DeltaEnumeratorStateCheckpointBuilder instance with TableProcessor state information.

    Definition Classes
    ChangesProcessor → TableProcessor
    Annotations
    @Override()
  21. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  22. def toString(): String
    Definition Classes
    AnyRef → Any
  23. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  24. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  25. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from ContinuousTableProcessor

Inherited from TableProcessorBase

Inherited from TableProcessor

Inherited from AnyRef

Inherited from Any

Ungrouped