Packages

class DeltaCommittable extends Serializable

Committable object that carries the information about files written to the file system during particular checkpoint interval.

As io.delta.flink.sink.DeltaSink implements both org.apache.flink.api.connector.sink.Committer and org.apache.flink.api.connector.sink.GlobalCommitter and then its committable must provide all metadata for committing data on both levels.

In order to commit data during org.apache.flink.api.connector.sink.Committer#commit information carried inside DeltaPendingFile are used. Next during org.apache.flink.api.connector.sink.GlobalCommitter#commit we are using both: metadata carried inside DeltaPendingFile and also transactional identifier constructed by application's unique id and checkpoint interval's id.

Lifecycle of instances of this class is as follows:

  • Every instance is created in io.delta.flink.sink.internal.writer.DeltaWriterBucket#prepareCommit method during a pre-commit phase.
  • When certain checkpointing barriers are reached then generated committables are snapshotted along with the rest of the application's state. See Flink's docs for details
See also

here

  • During commit phase every committable is first delivered to io.delta.flink.sink.internal.committer.DeltaCommitter#commit and then to io.delta.flink.sink.internal.committer.DeltaGlobalCommitter#combine methods when they are being committed.
  • If there's any failure of the app's execution then Flink may recover previously generated set of committables that may have not been committed. In such cases those recovered committables will be again passed to the committers' instance along with the new committables from the next checkpoint interval.
  • If checkpoint was successfull then committables from the given checkpoint interval are no longer recovered and exist only in the previously snapshotted states.
Linear Supertypes
Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DeltaCommittable
  2. Serializable
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new DeltaCommittable(deltaPendingFile: DeltaPendingFile, appId: String, checkpointId: Long)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. def getAppId(): String
  10. def getCheckpointId(): Long
  11. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. def getDeltaPendingFile(): DeltaPendingFile
  13. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  14. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  15. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  16. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  17. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  18. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  19. def toString(): String
    Definition Classes
    AnyRef → Any
  20. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  21. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  22. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped