Type Members

1.  class DeltaEnumeratorStateCheckpoint[SplitT <: DeltaSourceSplit] extends AnyRef

   A checkpoint of the current state of SplitEnumerator.

   A checkpoint of the current state of SplitEnumerator.

   It contains all necessary information need by SplitEnumerator to resume work after checkpoint recovery including currently pending splits that are not yet assigned and resume changes discovery task on Delta table in Boundedness#CONTINUOUS_UNBOUNDED mode

   During checkpoint, Flink will serialize this object and persist it in checkpoint location. During the recovery, Flink will deserialize this object from Checkpoint/Savepoint and will use it to recreate SplitEnumerator.

2.  class DeltaEnumeratorStateCheckpointBuilder[SplitT <: DeltaSourceSplit] extends AnyRef
3.  class DeltaPendingSplitsCheckpointSerializer[SplitT <: DeltaSourceSplit] extends SimpleVersionedSerializer[DeltaEnumeratorStateCheckpoint[SplitT]]

   A de/serializer for objects of class DeltaEnumeratorStateCheckpoint.

   A de/serializer for objects of class DeltaEnumeratorStateCheckpoint.

   This class provides methods for Flink core to serialize and deserialize DeltaPendingSplitsCheckpointSerializer objects.

   Serialization of DeltaPendingSplitsCheckpointSerializer object takes place during checkpoint operation.

   Deserialization of DeltaPendingSplitsCheckpointSerializer object takes place during recovering from checkpoint when SplitEnumerator is being recreated.

4.  class DeltaSourceSplit extends FileSourceSplit

   A SourceSplit that represents a Parquet file, or a region of a file.

   A SourceSplit that represents a Parquet file, or a region of a file.

   The split additionally has an offset and an end, which defines the region of the file represented by the split. For splits representing the while file, the offset is zero and the length is the file size.

   The split may furthermore have a "reader position", which is the checkpointed position from a reader previously reading this split. This position is null when the split is assigned from the enumerator to the readers, and is non-null when the reader's checkpoint their state in a file source split.

   This implementation extends a FileSourceSplit with Delta table partition information

5.  final class DeltaSourceSplitSerializer extends SimpleVersionedSerializer[DeltaSourceSplit]

   A de/serializer for objects of class DeltaSourceSplit.

   A de/serializer for objects of class DeltaSourceSplit.

   This class provides methods for Flink core to serialize and deserialize DeltaSourceSplit objects.

   Serialization of DeltaSourceSplit object takes place during checkpoint operation and when Splits are assigned to SourceReader by SplitEnumerator.

   Deserialization of DeltaSourceSplit object takes place during recovering from checkpoint and on a Task Manager nodes in Source Readers (SourceReader) after "receiving" assigned Split.