Packages

class DeltaPendingFile extends AnyRef

Wrapper class for InProgressFileWriter.PendingFileRecoverable object. This class carries the internal committable information to be used during the checkpoint/commit phase.

As similar to org.apache.flink.connector.file.sink.FileSink we need to carry InProgressFileWriter.PendingFileRecoverable information to perform "local" commit on file that the sink has written data to. However, as opposite to mentioned FileSink, in DeltaSink we need to perform also "global" commit to the io.delta.standalone.DeltaLog and for that additional file metadata must be provided. Hence, this class provides the required information for both types of commits by wrapping pending file and attaching file's metadata.

Lifecycle of instances of this class is as follows:

  • Instances of this class are being created inside io.delta.flink.sink.internal.writer.DeltaWriterBucket#closePartFile method every time when any in-progress is called to be closed. This happens either when some conditions for closing are met or at the end of every checkpoint interval during a pre-commit phase when we are closing all the open files in all buckets
  • Its life span holds only until the end of a checkpoint interval
  • During pre-commit phase (and after closing every in-progress files) every existing DeltaPendingFile instance is automatically transformed into a DeltaCommittable instance
Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DeltaPendingFile
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new DeltaPendingFile(partitionSpec: LinkedHashMap[String, String], fileName: String, pendingFile: PendingFileRecoverable, recordCount: Long, fileSize: Long, lastUpdateTime: Long)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  10. def getFileName(): String
  11. def getFileSize(): Long
  12. def getLastUpdateTime(): Long
  13. def getPartitionSpec(): LinkedHashMap[String, String]
  14. def getPendingFile(): PendingFileRecoverable
  15. def getRecordCount(): Long
  16. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  17. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  18. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  19. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  20. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  21. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  22. def toAddFile(): AddFile

    Converts DeltaPendingFile object to a AddFile object

    Converts DeltaPendingFile object to a AddFile object

    returns

    AddFile object generated from input

  23. def toString(): String
    Definition Classes
    DeltaPendingFile → AnyRef → Any
    Annotations
    @Override()
  24. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  25. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  26. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from AnyRef

Inherited from Any

Ungrouped