Packages

class DeltaWriter[IN] extends SinkWriter[IN, DeltaCommittable, DeltaWriterBucketState] with ProcessingTimeCallback

A SinkWriter implementation for io.delta.flink.sink.DeltaSink.

It writes data to and manages the different active buckets in the io.delta.flink.sink.DeltaSink.

Most of the logic for this class was sourced from FileWriter as the behaviour is very similar. The main differences are use of custom implementations for some member classes and also managing io.delta.standalone.DeltaLog transactional ids: DeltaWriter#appId and DeltaWriter#nextCheckpointId.

Lifecycle of instances of this class is as follows:

  • Every instance is being created via io.delta.flink.sink.DeltaSink#createWriter method
  • Writers' life span is the same as the application's (unless the worker node gets unresponding and the job manager needs to create a new instance to satisfy the parallelism)
  • Number of instances are managed globally by a job manager and this number is equal to the parallelism of the sink.
See also

Flink's parallel execution

Linear Supertypes
ProcessingTimeCallback, SinkWriter[IN, DeltaCommittable, DeltaWriterBucketState], AutoCloseable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DeltaWriter
  2. ProcessingTimeCallback
  3. SinkWriter
  4. AutoCloseable
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new DeltaWriter(basePath: Path, bucketAssigner: BucketAssigner[IN, String], bucketWriter: DeltaBulkBucketWriter[IN, String], rollingPolicy: CheckpointRollingPolicy[IN, String], outputFileConfig: OutputFileConfig, processingTimeService: ProcessingTimeService, metricGroup: MetricGroup, bucketCheckInterval: Long, appId: String, nextCheckpointId: Long)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. def close(): Unit

    Method for closing the writer, that it to say to dispose any in progress files.

    Method for closing the writer, that it to say to dispose any in progress files.

    Definition Classes
    DeltaWriter → AutoCloseable
    Annotations
    @Override()
  7. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  8. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  9. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  10. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. def initializeState(bucketStates: List[DeltaWriterBucketState]): Unit

    Initializes the state from snapshotted DeltaWriterBucketState.

    Initializes the state from snapshotted DeltaWriterBucketState.

    bucketStates

    the state holding recovered state about active buckets.

    Exceptions thrown

    IOException if anything goes wrong during retrieving the state or restoring/committing of any in-progress/pending part files

  13. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  14. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  15. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  16. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  17. def onProcessingTime(time: Long): Unit

    Method for getting current processing time ahd register timers.

    Method for getting current processing time ahd register timers.

    This method could be used e.g. to apply custom rolling file behaviour.

    Definition Classes
    DeltaWriter → ProcessingTimeCallback
    Annotations
    @Override()
  18. def prepareCommit(flush: Boolean): List[DeltaCommittable]

    This method prepares committables objects that will be passed to io.delta.flink.sink.internal.committer.DeltaCommitter and io.delta.flink.sink.internal.committer.DeltaGlobalCommitter to finalize the checkpoint interval and commit written files.

    This method prepares committables objects that will be passed to io.delta.flink.sink.internal.committer.DeltaCommitter and io.delta.flink.sink.internal.committer.DeltaGlobalCommitter to finalize the checkpoint interval and commit written files.

    Definition Classes
    DeltaWriter → SinkWriter
    Annotations
    @Override()
  19. def snapshotState(): List[DeltaWriterBucketState]

    Prepares the writer's state to be snapshotted between checkpoint intervals.

    Prepares the writer's state to be snapshotted between checkpoint intervals.

    Definition Classes
    DeltaWriter → SinkWriter
    Annotations
    @Override()
  20. def snapshotState(arg0: Long): List[DeltaWriterBucketState]
    Definition Classes
    SinkWriter
    Annotations
    @throws( classOf[java.io.IOException] )
  21. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  22. def toString(): String
    Definition Classes
    AnyRef → Any
  23. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  24. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  25. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  26. def write(element: IN, context: Context): Unit

    A proxy method that forwards the incoming event to the correct DeltaWriterBucket instance.

    A proxy method that forwards the incoming event to the correct DeltaWriterBucket instance.

    element

    incoming stream event

    context

    context for getting additional data about input event

    Definition Classes
    DeltaWriter → SinkWriter
    Annotations
    @Override()
  27. def writeWatermark(arg0: Watermark): Unit
    Definition Classes
    SinkWriter
    Annotations
    @throws( classOf[java.io.IOException] ) @throws( ... )

Inherited from ProcessingTimeCallback

Inherited from SinkWriter[IN, DeltaCommittable, DeltaWriterBucketState]

Inherited from AutoCloseable

Inherited from AnyRef

Inherited from Any

Ungrouped