Type Members

1.  class DeltaCommittable extends Serializable

   Committable object that carries the information about files written to the file system during particular checkpoint interval.

   Committable object that carries the information about files written to the file system during particular checkpoint interval.

   As io.delta.flink.sink.DeltaSink implements both org.apache.flink.api.connector.sink.Committer and org.apache.flink.api.connector.sink.GlobalCommitter and then its committable must provide all metadata for committing data on both levels.

   In order to commit data during org.apache.flink.api.connector.sink.Committer#commit information carried inside DeltaPendingFile are used. Next during org.apache.flink.api.connector.sink.GlobalCommitter#commit we are using both: metadata carried inside DeltaPendingFile and also transactional identifier constructed by application's unique id and checkpoint interval's id.

   Lifecycle of instances of this class is as follows:

   • Every instance is created in io.delta.flink.sink.internal.writer.DeltaWriterBucket#prepareCommit method during a pre-commit phase.
   • When certain checkpointing barriers are reached then generated committables are snapshotted along with the rest of the application's state. See Flink's docs for details

   See also

   here

   • During commit phase every committable is first delivered to io.delta.flink.sink.internal.committer.DeltaCommitter#commit and then to io.delta.flink.sink.internal.committer.DeltaGlobalCommitter#combine methods when they are being committed.
   • If there's any failure of the app's execution then Flink may recover previously generated set of committables that may have not been committed. In such cases those recovered committables will be again passed to the committers' instance along with the new committables from the next checkpoint interval.
   • If checkpoint was successfull then committables from the given checkpoint interval are no longer recovered and exist only in the previously snapshotted states.

2.  class DeltaCommittableSerializer extends SimpleVersionedSerializer[DeltaCommittable]

   Versioned serializer for DeltaCommittable.

3.  class DeltaGlobalCommittable extends AnyRef

   Simple wrapper class for a collection of DeltaCommittable instances.

   Simple wrapper class for a collection of DeltaCommittable instances.

   This class is provided to comply with the org.apache.flink.api.connector.sink.GlobalCommitter interfaces' structure. It's only purpose is to wrap DeltaCommittable collection during io.delta.flink.sink.internal.committer.DeltaGlobalCommitter#combine method that will be further flattened and processed inside io.delta.flink.sink.internal.committer.DeltaGlobalCommitter#commit method.

   Lifecycle of instances of this class is as follows:

   • Every instance is created in io.delta.flink.sink.internal.committer.DeltaGlobalCommitter#combine method during a global commit phase.
   • When certain checkpointing barriers are reached then generated committables are snapshotted along with the rest of the application's state. See Flink's docs for details

   See also

   here

   • Every DeltaGlobalCommittable instance is delivered to io.delta.flink.sink.internal.committer.DeltaGlobalCommitter#combine method when they are being committed to a io.delta.standalone.DeltaLog.
   • If there's any failure of the app's execution then Flink may recover previously generated set of committables that may have not been committed. In such cases those recovered committables will be again passed to the org.apache.flink.api.connector.sink.GlobalCommitter instance along with the new set of committables from the next checkpoint interval.
   • If checkpoint was successfull then committables from the given checkpoint interval are no longer recovered and exist only in the previously snapshotted states.

4.  class DeltaGlobalCommittableSerializer extends SimpleVersionedSerializer[DeltaGlobalCommittable]

   Versioned serializer for DeltaGlobalCommittable.