trait DeltaSourceCDCSupport extends AnyRef
Helper functions for CDC-specific handling for DeltaSource.
- Self Type
- DeltaSource
- Alphabetic
- By Inheritance
- DeltaSourceCDCSupport
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Type Members
- class IndexedChangeFileSeq extends AnyRef
This class represents an iterator of Change metadata(AddFile, RemoveFile, AddCDCFile) for a particular version.
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @native()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable])
- def getCDCFileChangesAndCreateDataFrame(startVersion: Long, startIndex: Long, isInitialSnapshot: Boolean, endOffset: DeltaSourceOffset): DataFrame
Get the changes from startVersion, startIndex to the end for CDC case.
Get the changes from startVersion, startIndex to the end for CDC case. We need to call CDCReader to get the CDC DataFrame.
- startVersion
- calculated starting version
- startIndex
- calculated starting index
- isInitialSnapshot
- whether the stream has to return the initial snapshot or not
- endOffset
- Offset that signifies the end of the stream.
- returns
the DataFrame containing the file changes (AddFile, RemoveFile, AddCDCFile)
- Attributes
- protected
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def getFileChangesForCDC(fromVersion: Long, fromIndex: Long, isInitialSnapshot: Boolean, limits: Option[AdmissionLimits], endOffset: Option[DeltaSourceOffset], verifyMetadataAction: Boolean = true): Iterator[(Long, Iterator[IndexedFile], Option[CommitInfo])]
Get the changes starting from (fromVersion, fromIndex).
Get the changes starting from (fromVersion, fromIndex). fromVersion is included. It returns an iterator of (log_version, fileActions, Optional[CommitInfo]). The commit info is needed later on so that the InCommitTimestamp of the log files can be determined.
If verifyMetadataAction = true, we will break the stream when we detect any read-incompatible metadata changes.
- Attributes
- protected
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()