Packages

c

io.delta.flink.source.internal.state

DeltaSourceSplit

class DeltaSourceSplit extends FileSourceSplit

A SourceSplit that represents a Parquet file, or a region of a file.

The split additionally has an offset and an end, which defines the region of the file represented by the split. For splits representing the while file, the offset is zero and the length is the file size.

The split may furthermore have a "reader position", which is the checkpointed position from a reader previously reading this split. This position is null when the split is assigned from the enumerator to the readers, and is non-null when the reader's checkpoint their state in a file source split.

This implementation extends a FileSourceSplit with Delta table partition information

Linear Supertypes
FileSourceSplit, Serializable, SourceSplit, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DeltaSourceSplit
  2. FileSourceSplit
  3. Serializable
  4. SourceSplit
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new DeltaSourceSplit(partitionValues: Map[String, String], id: String, filePath: Path, offset: Long, length: Long, hostnames: Array[String], readerPosition: CheckpointedPosition)

    Constructs a split with host information and reader position restored from checkpoint.

    Constructs a split with host information and reader position restored from checkpoint.

    The hostnames parameter provides information about the names of the hosts storing this range of the file. Empty, if no host information is available. Host information is typically only available on a specific file systems, like HDFS.

    partitionValues

    The Delta partition column to partition value map that should be used for underlying Parquet File.

    id

    The unique ID of this source split.

    filePath

    The path to the Parquet file that this splits represents.

    offset

    The start (inclusive) of the split's rage in the Parquet file, in bytes.

    length

    The number of bytes in the split (starting from the offset)

    hostnames

    The hostnames of the nodes storing the split's file range.

    readerPosition

    The reader position in bytes recovered from a checkpoint.

  2. new DeltaSourceSplit(partitionValues: Map[String, String], id: String, filePath: Path, offset: Long, length: Long, hostnames: <repeated...>[String])

    Constructs a split with host information and no reader position.

    Constructs a split with host information and no reader position.

    The hostnames provides information about the names of the hosts is storing this range of the file. Empty, if no host information is available. Host information is typically only available on a specific file systems, like HDFS.

    partitionValues

    The Delta partition column to partition value map that should be used for underlying Parquet File.

    id

    The unique ID of this source split.

    filePath

    The path to the Parquet file that this splits represents.

    offset

    The start (inclusive) of the split's rage in the Parquet file, in bytes.

    length

    The number of bytes in the split (starting from the offset)

    hostnames

    The hostnames of the nodes storing the split's file range.

  3. new DeltaSourceSplit(partitionValues: Map[String, String], id: String, filePath: Path, offset: Long, length: Long)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. def fileModificationTime(): Long
    Definition Classes
    FileSourceSplit
  9. def fileSize(): Long
    Definition Classes
    FileSourceSplit
  10. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  11. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. def getPartitionValues(): Map[String, String]

    returns

    an unmodifiable Map of Delta Table Partition columns and values.

  13. def getReaderPosition(): Optional[CheckpointedPosition]
    Definition Classes
    FileSourceSplit
  14. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  15. def hostnames(): Array[String]
    Definition Classes
    FileSourceSplit
  16. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  17. def length(): Long
    Definition Classes
    FileSourceSplit
  18. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  19. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  20. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  21. def offset(): Long
    Definition Classes
    FileSourceSplit
  22. def path(): Path
    Definition Classes
    FileSourceSplit
  23. def splitId(): String
    Definition Classes
    FileSourceSplit → SourceSplit
  24. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  25. def toString(): String
    Definition Classes
    FileSourceSplit → AnyRef → Any
  26. def updateWithCheckpointedPosition(position: CheckpointedPosition): DeltaSourceSplit
    Definition Classes
    DeltaSourceSplit → FileSourceSplit
    Annotations
    @Override()
  27. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  28. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  29. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from FileSourceSplit

Inherited from Serializable

Inherited from SourceSplit

Inherited from AnyRef

Inherited from Any

Ungrouped