object UniversalFormat extends DeltaLogging
Utils to validate the Universal Format (UniForm) Delta feature (NOT a table feature).
The UniForm Delta feature governs and implements the actual conversion of Delta metadata into other formats.
UniForm supports both Iceberg and Hudi. When delta.universalFormat.enabledFormats contains
"iceberg", we say that Universal Format (Iceberg) is enabled. When it contains "hudi", we say
that Universal Format (Hudi) is enabled.
enforceInvariantsAndDependencies ensures that all of UniForm's requirements for the specified format are met (e.g. for 'iceberg' that IcebergCompatV1 or V2 is enabled). It doesn't verify that its nested requirements are met (e.g. IcebergCompat's requirements, like Column Mapping). That is the responsibility of format-specific validations such as IcebergCompatV1.enforceInvariantsAndDependencies and IcebergCompatV2.enforceInvariantsAndDependencies.
Note that UniForm (Iceberg) depends on IcebergCompat, but IcebergCompat does not depend on or require UniForm (Iceberg). It is perfectly valid for a Delta table to have IcebergCompatV1 or V2 enabled but UniForm (Iceberg) not enabled.
- Alphabetic
- By Inheritance
- UniversalFormat
- DeltaLogging
- DatabricksLogging
- DeltaProgressReporter
- LoggingShims
- Logging
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Type Members
- implicit class LogStringContext extends AnyRef
- Definition Classes
- LoggingShims
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- val HUDI_FORMAT: String
- val ICEBERG_FORMAT: String
- val ICEBERG_TABLE_TYPE_KEY: String
- val SUPPORTED_FORMATS: Set[String]
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @native()
- def deltaAssert(check: => Boolean, name: String, msg: String, deltaLog: DeltaLog = null, data: AnyRef = null, path: Option[Path] = None): Unit
Helper method to check invariants in Delta code.
Helper method to check invariants in Delta code. Fails when running in tests, records a delta assertion event and logs a warning otherwise.
- Attributes
- protected
- Definition Classes
- DeltaLogging
- def enforceDependenciesInConfiguration(configuration: Map[String, String], snapshot: Snapshot): Map[String, String]
This method is used to build UniForm metadata dependencies closure.
This method is used to build UniForm metadata dependencies closure. It checks configuration conflicts and adds missing properties. It will call enforceIcebergInvariantsAndDependencies to perform the actual check.
- configuration
the original metadata configuration.
- returns
updated configuration if any changes are required, otherwise the original configuration.
- def enforceHudiDependencies(newestMetadata: Metadata, snapshot: Snapshot): Any
If you are enabling Hudi, this method ensures that Deletion Vectors are not enabled.
If you are enabling Hudi, this method ensures that Deletion Vectors are not enabled. New conditions may be added here in the future to make sure the source is compatible with Hudi.
- newestMetadata
the newest metadata
- snapshot
current snapshot
- returns
N/A, throws exception if condition is not met
- def enforceIcebergInvariantsAndDependencies(snapshot: Snapshot, newestProtocol: Protocol, newestMetadata: Metadata, operation: Option[Operation], actions: Seq[Action]): (Option[Protocol], Option[Metadata])
If you are enabling Universal Format (Iceberg), this method ensures that at least one of IcebergCompat is enabled.
If you are enabling Universal Format (Iceberg), this method ensures that at least one of IcebergCompat is enabled. If you are disabling Universal Format (Iceberg), this method will leave the current IcebergCompat version untouched.
- returns
tuple of options of (updatedProtocol, updatedMetadata). For either action, if no updates need to be applied, will return None.
- def enforceInvariantsAndDependencies(snapshot: Snapshot, newestProtocol: Protocol, newestMetadata: Metadata, operation: Option[Operation], actions: Seq[Action]): (Option[Protocol], Option[Metadata])
Expected to be called after the newest metadata and protocol have been ~ finalized.
Expected to be called after the newest metadata and protocol have been ~ finalized.
- returns
tuple of options of (updatedProtocol, updatedMetadata). For either action, if no updates need to be applied, will return None.
- def enforceSupportInCatalog(table: CatalogTable, metadata: Metadata): Option[CatalogTable]
Update CatalogTable to mark it readable by other table readers (iceberg for now).
Update CatalogTable to mark it readable by other table readers (iceberg for now). This method ensures 'table_type' = 'ICEBERG' when uniform is enabled, and ensure table_type is not 'ICEBERG' when uniform is not enabled If the key has other values than 'ICEBERG', this method will not touch it for compatibility
- table
catalogTable before change
- metadata
snapshot metadata
- returns
the converted catalog, or None if no change is made
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable])
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def getCommonTags(deltaLog: DeltaLog, tahoeId: String): Map[TagDefinition, String]
- Definition Classes
- DeltaLogging
- def getErrorData(e: Throwable): Map[String, Any]
- Definition Classes
- DeltaLogging
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def hudiEnabled(properties: Map[String, String]): Boolean
- def hudiEnabled(metadata: Metadata): Boolean
- def icebergEnabled(properties: Map[String, String]): Boolean
- def icebergEnabled(metadata: Metadata): Boolean
- def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
- Attributes
- protected
- Definition Classes
- Logging
- def initializeLogIfNecessary(isInterpreter: Boolean): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def isCreatingOrReorgTable(op: Option[Operation]): Boolean
Check if the operation is CREATE/REPLACE TABLE or REORG UPGRADE UNIFORM commands.
Check if the operation is CREATE/REPLACE TABLE or REORG UPGRADE UNIFORM commands.
- op
the delta operation to be checked.
- returns
whether the operation is create or reorg.
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- def isReorgUpgradeUniform(op: Option[Operation]): Boolean
Check if the operation is REORG UPGRADE UNIFORM command.
Check if the operation is REORG UPGRADE UNIFORM command.
- op
the delta operation to be checked.
- returns
whether the operation is REORG UPGRADE UNIFORM.
- def isTraceEnabled(): Boolean
- Attributes
- protected
- Definition Classes
- Logging
- def log: Logger
- Attributes
- protected
- Definition Classes
- Logging
- def logConsole(line: String): Unit
- Definition Classes
- DatabricksLogging
- def logDebug(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- LoggingShims
- def logDebug(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- LoggingShims
- def logDebug(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- LoggingShims
- def logError(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- LoggingShims
- def logError(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- LoggingShims
- def logInfo(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- LoggingShims
- def logInfo(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logName: String
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- LoggingShims
- def logTrace(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- LoggingShims
- def logTrace(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- LoggingShims
- def logWarning(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- LoggingShims
- def logWarning(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- def recordDeltaEvent(deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty, data: AnyRef = null, path: Option[Path] = None): Unit
Used to record the occurrence of a single event or report detailed, operation specific statistics.
Used to record the occurrence of a single event or report detailed, operation specific statistics.
- path
Used to log the path of the delta table when
deltaLogis null.
- Attributes
- protected
- Definition Classes
- DeltaLogging
- def recordDeltaOperation[A](deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: => A): A
Used to report the duration as well as the success or failure of an operation on a
deltaLog.Used to report the duration as well as the success or failure of an operation on a
deltaLog.- Attributes
- protected
- Definition Classes
- DeltaLogging
- def recordDeltaOperationForTablePath[A](tablePath: String, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: => A): A
Used to report the duration as well as the success or failure of an operation on a
tahoePath.Used to report the duration as well as the success or failure of an operation on a
tahoePath.- Attributes
- protected
- Definition Classes
- DeltaLogging
- def recordEvent(metric: MetricDefinition, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit
- Definition Classes
- DatabricksLogging
- def recordFrameProfile[T](group: String, name: String)(thunk: => T): T
- Attributes
- protected
- Definition Classes
- DeltaLogging
- def recordOperation[S](opType: OpType, opTarget: String = null, extraTags: Map[TagDefinition, String], isSynchronous: Boolean = true, alwaysRecordStats: Boolean = false, allowAuthTags: Boolean = false, killJvmIfStuck: Boolean = false, outputMetric: MetricDefinition = METRIC_OPERATION_DURATION, silent: Boolean = true)(thunk: => S): S
- Definition Classes
- DatabricksLogging
- def recordProductEvent(metric: MetricDefinition with CentralizableMetric, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit
- Definition Classes
- DatabricksLogging
- def recordProductUsage(metric: MetricDefinition with CentralizableMetric, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit
- Definition Classes
- DatabricksLogging
- def recordUsage(metric: MetricDefinition, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit
- Definition Classes
- DatabricksLogging
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- def withStatusCode[T](statusCode: String, defaultMessage: String, data: Map[String, Any] = Map.empty)(body: => T): T
Report a log to indicate some command is running.
Report a log to indicate some command is running.
- Definition Classes
- DeltaProgressReporter