Packages

class DeltaCommitter extends Committer[DeltaCommittable]

Committer implementation for DeltaSink.

This committer is responsible for taking staged part-files, i.e. part-files in "pending" state, created by the io.delta.flink.sink.internal.writer.DeltaWriter and put them in "finished" state ready to be committed to the DeltaLog during "global" commit.

This class behaves almost in the same way as its equivalent org.apache.flink.connector.file.sink.committer.FileCommitter in the org.apache.flink.connector.file.sink.FileSink. The only differences are:

  • use of the DeltaCommittable instead of org.apache.flink.connector.file.sink.FileSinkCommittable
  • some simplifications for the committable's internal information and commit behaviour. In particular in DeltaCommitter#commit method we do not take care of any inprogress file's state (as opposite to org.apache.flink.connector.file.sink.committer.FileCommitter#commit because in DeltaWriter#prepareCommit we always roll all of the in-progress files. Valid note here is that's also the default org.apache.flink.connector.file.sink.FileSink's behaviour for all of the bulk formats (Parquet included).

Lifecycle of instances of this class is as follows:

  • Instances of this class are being created during a commit stage
  • For every DeltaWriter object there is only one of corresponding DeltaCommitter created, thus the number of created instances is equal to the parallelism of the application's sink
  • Every instance exists only during given commit stage after finishing particular checkpoint interval. Despite being bundled to a finish phase of a checkpoint interval a single instance of DeltaCommitter may process committables from multiple checkpoints intervals (it happens e.g. when there was a app's failure and Flink has recovered committables from previous commit stage to be re-committed.
Linear Supertypes
Committer[DeltaCommittable], AutoCloseable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DeltaCommitter
  2. Committer
  3. AutoCloseable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new DeltaCommitter(bucketWriter: BucketWriter[_, _])

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. def close(): Unit
    Definition Classes
    DeltaCommitter → AutoCloseable
    Annotations
    @Override()
  7. def commit(committables: List[DeltaCommittable]): List[DeltaCommittable]

    This method is responsible for "committing" files locally.

    This method is responsible for "committing" files locally.

    "Local" commit in our case means the same as in org.apache.flink.connector.file.sink.committer.FileCommitter#commit, namely it's the simple process of renaming the hidden file to make it visible and removing from the name some 'in-progress file' marker. For details see internal interfaces in org.apache.flink.streaming.api.functions.sink.filesystem.BucketWriter.

    committables

    list of committables. May contain committables from multiple checkpoint intervals

    returns

    always empty list as we do not allow or expect any retry behaviour

    Definition Classes
    DeltaCommitter → Committer
    Annotations
    @Override()
    Exceptions thrown

    IOException if committing files (e.g. I/O errors occurs)

  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  10. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  11. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  13. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  14. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  15. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  16. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  17. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  18. def toString(): String
    Definition Classes
    AnyRef → Any
  19. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  20. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  21. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from Committer[DeltaCommittable]

Inherited from AutoCloseable

Inherited from AnyRef

Inherited from Any

Ungrouped