abstract class UniversalFormatConverter extends AnyRef
Class to facilitate the conversion of Delta into other table formats.
- Alphabetic
- By Inheritance
- UniversalFormatConverter
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new UniversalFormatConverter(spark: SparkSession)
Abstract Value Members
- abstract def convertSnapshot(snapshotToConvert: Snapshot, catalogTable: CatalogTable): Option[(Long, Long)]
Perform a blocking conversion for the given catalogTable
Perform a blocking conversion for the given catalogTable
- snapshotToConvert
the snapshot that needs to be converted to Iceberg
- catalogTable
the catalogTable this conversion targets.
- returns
Converted Delta version and commit timestamp
- abstract def convertSnapshot(snapshotToConvert: Snapshot, txn: OptimisticTransactionImpl): Option[(Long, Long)]
Perform a blocking conversion when performing an OptimisticTransaction on a delta table.
Perform a blocking conversion when performing an OptimisticTransaction on a delta table.
- snapshotToConvert
the snapshot that needs to be converted to Iceberg
- txn
the transaction that triggers the conversion. Used as a hint to avoid recomputing old metadata. It must contain the catalogTable this conversion targets.
- returns
Converted Delta version and commit timestamp
- abstract def enqueueSnapshotForConversion(snapshotToConvert: Snapshot, txn: OptimisticTransactionImpl): Unit
Perform an asynchronous conversion.
Perform an asynchronous conversion.
This will start an async job to run the conversion, unless there already is an async conversion running for this table. In that case, it will queue up the provided snapshot to be run after the existing job completes.
- abstract def loadLastDeltaVersionConverted(snapshot: Snapshot, table: CatalogTable): Option[Long]
Fetch the delta version corresponding to the latest conversion.
Fetch the delta version corresponding to the latest conversion.
- snapshot
the snapshot to be converted
- table
the catalogTable with info of previous conversions
- returns
None if no previous conversion found
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @native()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable])
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()