case class PersistedMetadata(tableId: String, deltaCommitVersion: Long, dataSchemaJson: String, partitionSchemaJson: String, sourceMetadataPath: String, tableConfigurations: Option[Map[String, String]] = None, protocolJson: Option[String] = None, previousMetadataSeqNum: Option[Long] = None) extends PartitionAndDataSchema with Product with Serializable

A PersistedMetadata is an entry in Delta streaming source schema log, which can be used to read data files during streaming.

tableId

Delta table id

deltaCommitVersion

Delta commit version in which this change is captured. It does not necessarily have to be the commit when there's an actual change, e.g. during initialization. The invariant is that the metadata must be read-compatible with the table snapshot at this version.

dataSchemaJson

Full schema json

partitionSchemaJson

Partition schema json

sourceMetadataPath

The checkpoint path that is unique to each source.

tableConfigurations

The configurations of the table inside the metadata when the schema change was detected. It is used to correctly create the right file format when we use a particular schema to read. Default to None for backward compatibility.

protocolJson

JSON of the protocol change if any. Default to None for backward compatibility.

previousMetadataSeqNum

When defined, it points to the batch ID / seq num for the previous metadata in the log sequence. It is used when we could not reliably tell if the currentBatchId - 1 is indeed the previous schema evolution, e.g. when we are merging consecutive schema changes during the analysis phase and we are appending an extra schema after the merge to the log. Default to None for backward compatibility.

Linear Supertypes
Serializable, Serializable, Product, Equals, PartitionAndDataSchema, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. PersistedMetadata
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. PartitionAndDataSchema
  7. AnyRef
  8. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new PersistedMetadata(tableId: String, deltaCommitVersion: Long, dataSchemaJson: String, partitionSchemaJson: String, sourceMetadataPath: String, tableConfigurations: Option[Map[String, String]] = None, protocolJson: Option[String] = None, previousMetadataSeqNum: Option[Long] = None)

    tableId

    Delta table id

    deltaCommitVersion

    Delta commit version in which this change is captured. It does not necessarily have to be the commit when there's an actual change, e.g. during initialization. The invariant is that the metadata must be read-compatible with the table snapshot at this version.

    dataSchemaJson

    Full schema json

    partitionSchemaJson

    Partition schema json

    sourceMetadataPath

    The checkpoint path that is unique to each source.

    tableConfigurations

    The configurations of the table inside the metadata when the schema change was detected. It is used to correctly create the right file format when we use a particular schema to read. Default to None for backward compatibility.

    protocolJson

    JSON of the protocol change if any. Default to None for backward compatibility.

    previousMetadataSeqNum

    When defined, it points to the batch ID / seq num for the previous metadata in the log sequence. It is used when we could not reliably tell if the currentBatchId - 1 is indeed the previous schema evolution, e.g. when we are merging consecutive schema changes during the analysis phase and we are appending an extra schema after the merge to the log. Default to None for backward compatibility.

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. lazy val dataSchema: StructType
    Definition Classes
    PersistedMetadata → PartitionAndDataSchema
    Annotations
    @JsonIgnore()
  7. val dataSchemaJson: String
  8. val deltaCommitVersion: Long
  9. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  10. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  11. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  13. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  14. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  15. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  16. lazy val partitionSchema: StructType
    Definition Classes
    PersistedMetadata → PartitionAndDataSchema
    Annotations
    @JsonIgnore()
  17. val partitionSchemaJson: String
  18. val previousMetadataSeqNum: Option[Long]
  19. lazy val protocol: Option[Protocol]
    Annotations
    @JsonIgnore()
  20. val protocolJson: Option[String]
  21. val sourceMetadataPath: String
  22. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  23. val tableConfigurations: Option[Map[String, String]]
  24. val tableId: String
  25. def validateAgainstSnapshot(snapshot: SnapshotDescriptor): Unit
  26. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  27. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  28. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from PartitionAndDataSchema

Inherited from AnyRef

Inherited from Any

Ungrouped