Packages

class DeltaInProgressPart[IN] extends AnyRef

Wrapper class for part files in the io.delta.flink.sink.DeltaSink. Part files are files that are currently "opened" for writing new data. Similar behaviour might be observed in the org.apache.flink.connector.file.sink.FileSink however as opposite to the FileSink, in DeltaSink we need to keep the name of the file attached to the opened file in order to be further able to transform DeltaInProgressPart instance into DeltaPendingFile instance and finally to commit the written file to the io.delta.standalone.DeltaLog during global commit phase.

Additionally, we need a custom implementation of DeltaBulkPartWriter as a workaround for getting actual file size (what is currently not possible for bulk formats when operating on an interface level of PartFileInfo, see DeltaBulkPartWriter for details).

Lifecycle of instances of this class is as follows:

  • Instances of this class are being created inside io.delta.flink.sink.internal.writer.DeltaWriterBucket#rollPartFile method every time a bucket processes the first event or if the previously opened file met conditions for rolling (e.g. size threshold)
  • It's life span holds as long as the underlying file stays in an in-progress state (so until it's "rolled"), but no longer then single checkpoint interval.
  • During pre-commit phase every existing DeltaInProgressPart instance is automatically transformed ("rolled") into a DeltaPendingFile instance
Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DeltaInProgressPart
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new DeltaInProgressPart(fileName: String, bulkPartWriter: DeltaBulkPartWriter[IN, String])

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. def getBulkPartWriter(): DeltaBulkPartWriter[IN, String]
  10. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. def getFileName(): String
  12. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  13. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  14. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  15. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  16. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  17. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  18. def toString(): String
    Definition Classes
    AnyRef → Any
  19. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  20. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  21. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from AnyRef

Inherited from Any

Ungrouped