object ClusteredTableUtils extends ClusteredTableUtilsBase
- Alphabetic
- By Inheritance
- ClusteredTableUtils
- ClusteredTableUtilsBase
- DeltaLogging
- DatabricksLogging
- DeltaProgressReporter
- LoggingShims
- Logging
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Type Members
- implicit class LogStringContext extends AnyRef
- Definition Classes
- LoggingShims
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- val PROP_CLUSTERING_COLUMNS: String
- Definition Classes
- ClusteredTableUtilsBase
- def areClusteringColumnsInStatsSchema(snapshot: Snapshot, logicalClusteringColumns: Seq[String]): Boolean
Returns true if stats will be collected for all clustering columns.
Returns true if stats will be collected for all clustering columns.
- Definition Classes
- ClusteredTableUtilsBase
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @native()
- def clusteringProvider: String
The clustering implementation name for AddFile.clusteringProvider
The clustering implementation name for AddFile.clusteringProvider
- Definition Classes
- ClusteredTableUtilsBase
- def createDomainMetadata(clusteringColumns: Seq[ClusteringColumn]): DomainMetadata
Create a DomainMetadata action to store clustering columns.
Create a DomainMetadata action to store clustering columns.
- Definition Classes
- ClusteredTableUtilsBase
- def deltaAssert(check: => Boolean, name: String, msg: String, deltaLog: DeltaLog = null, data: AnyRef = null, path: Option[Path] = None): Unit
Helper method to check invariants in Delta code.
Helper method to check invariants in Delta code. Fails when running in tests, records a delta assertion event and logs a warning otherwise.
- Attributes
- protected
- Definition Classes
- DeltaLogging
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable])
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def getClusterBySpecOptional(snapshot: Snapshot): Option[ClusterBySpec]
Returns an optional ClusterBySpec from the given Snapshot.
Returns an optional ClusterBySpec from the given Snapshot.
- Definition Classes
- ClusteredTableUtilsBase
- def getClusterBySpecOptional(table: CatalogTable): Option[ClusterBySpec]
Returns an optional ClusterBySpec from the given CatalogTable.
Returns an optional ClusterBySpec from the given CatalogTable.
- Definition Classes
- ClusteredTableUtilsBase
- def getClusteringColumnsAsProperty(snapshot: Snapshot): Option[(String, String)]
Extract clustering columns from a given snapshot.
Extract clustering columns from a given snapshot.
- Definition Classes
- ClusteredTableUtilsBase
- def getClusteringColumnsAsProperty(maybeClusterBySpec: Option[ClusterBySpec]): Option[(String, String)]
Extract clustering columns from ClusterBySpec.
Extract clustering columns from ClusterBySpec.
- maybeClusterBySpec
optional ClusterBySpec. If it's empty, will return the original properties.
- returns
an optional pair with clustering columns.
- Definition Classes
- ClusteredTableUtilsBase
- def getClusteringColumnsOptional(snapshot: Snapshot): Option[Seq[ClusteringColumn]]
Extract ClusteringColumns from a given snapshot.
Extract ClusteringColumns from a given snapshot. Return None if the clustering domain metadata is missing.
- Definition Classes
- ClusteredTableUtilsBase
- def getClusteringDomainMetadata(snapshot: Snapshot): Seq[DomainMetadata]
Extract DomainMetadata for storing clustering columns from a given snapshot.
Extract DomainMetadata for storing clustering columns from a given snapshot. It returns clustering domain metadata if exists. Return empty if the clustering domain metadata is missing.
- Definition Classes
- ClusteredTableUtilsBase
- def getClusteringDomainMetadataForAlterTableClusterBy(newLogicalClusteringColumns: Seq[String], txn: OptimisticTransaction): Seq[DomainMetadata]
Create new clustering DomainMetadata actions given updated column names for 'ALTER TABLE ...
Create new clustering DomainMetadata actions given updated column names for 'ALTER TABLE ... CLUSTER BY'.
- Definition Classes
- ClusteredTableUtilsBase
- def getCommonTags(deltaLog: DeltaLog, tahoeId: String): Map[TagDefinition, String]
- Definition Classes
- DeltaLogging
- def getDomainMetadataFromTransaction(clusterBySpecOpt: Option[ClusterBySpec], txn: OptimisticTransaction): Seq[DomainMetadata]
Returns DomainMetadata action to store clustering columns.
Returns DomainMetadata action to store clustering columns. If clusterBySpecOpt is not empty (clustering columns are specified by CLUSTER BY), it creates the domain metadata based on the clustering columns. Otherwise (CLUSTER BY is not specified for REPLACE TABLE), it creates the domain metadata with empty clustering columns if a clustering domain exists.
This is used for CREATE TABLE and REPLACE TABLE.
- Definition Classes
- ClusteredTableUtilsBase
- def getErrorData(e: Throwable): Map[String, Any]
- Definition Classes
- DeltaLogging
- def getLogicalClusteringColumnNames(txn: OptimisticTransaction, actionsToCommit: Seq[Action]): Option[Seq[String]]
Extract the logical clustering column names from the to-be committed domain metadata action.
Extract the logical clustering column names from the to-be committed domain metadata action.
- txn
the transaction being used to commit the actions.
- actionsToCommit
the actions to be committed.
- returns
optional logical clustering column names.
- Definition Classes
- ClusteredTableUtilsBase
- def getMatchingMetadataDomain(clusteringColumns: Seq[ClusteringColumn], existingDomainMetadata: Seq[DomainMetadata]): MatchingMetadataDomain
Returns a sequence of DomainMetadata actions to update the existing domain metadata with the given clustering columns.
Returns a sequence of DomainMetadata actions to update the existing domain metadata with the given clustering columns.
This is mainly used for REPLACE TABLE and RESTORE TABLE.
- Definition Classes
- ClusteredTableUtilsBase
- def getTableFeatureProperties(existingProperties: Map[String, String]): Map[String, String]
Returns table feature properties that's required to create a clustered table.
Returns table feature properties that's required to create a clustered table.
- existingProperties
Table properties set by the user when creating a clustered table.
- Definition Classes
- ClusteredTableUtilsBase
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
- Attributes
- protected
- Definition Classes
- Logging
- def initializeLogIfNecessary(isInterpreter: Boolean): Unit
- Attributes
- protected
- Definition Classes
- Logging
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- def isSupported(protocol: Protocol): Boolean
Returns whether the protocol version supports the Liquid table feature.
Returns whether the protocol version supports the Liquid table feature.
- Definition Classes
- ClusteredTableUtilsBase
- def isTraceEnabled(): Boolean
- Attributes
- protected
- Definition Classes
- Logging
- def log: Logger
- Attributes
- protected
- Definition Classes
- Logging
- def logConsole(line: String): Unit
- Definition Classes
- DatabricksLogging
- def logDebug(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- LoggingShims
- def logDebug(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- LoggingShims
- def logDebug(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- LoggingShims
- def logError(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- LoggingShims
- def logError(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- LoggingShims
- def logInfo(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- LoggingShims
- def logInfo(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logName: String
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- LoggingShims
- def logTrace(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- LoggingShims
- def logTrace(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(entry: LogEntry, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- LoggingShims
- def logWarning(entry: LogEntry): Unit
- Attributes
- protected
- Definition Classes
- LoggingShims
- def logWarning(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- def recordDeltaEvent(deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty, data: AnyRef = null, path: Option[Path] = None): Unit
Used to record the occurrence of a single event or report detailed, operation specific statistics.
Used to record the occurrence of a single event or report detailed, operation specific statistics.
- path
Used to log the path of the delta table when
deltaLogis null.
- Attributes
- protected
- Definition Classes
- DeltaLogging
- def recordDeltaOperation[A](deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: => A): A
Used to report the duration as well as the success or failure of an operation on a
deltaLog.Used to report the duration as well as the success or failure of an operation on a
deltaLog.- Attributes
- protected
- Definition Classes
- DeltaLogging
- def recordDeltaOperationForTablePath[A](tablePath: String, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: => A): A
Used to report the duration as well as the success or failure of an operation on a
tahoePath.Used to report the duration as well as the success or failure of an operation on a
tahoePath.- Attributes
- protected
- Definition Classes
- DeltaLogging
- def recordEvent(metric: MetricDefinition, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit
- Definition Classes
- DatabricksLogging
- def recordFrameProfile[T](group: String, name: String)(thunk: => T): T
- Attributes
- protected
- Definition Classes
- DeltaLogging
- def recordOperation[S](opType: OpType, opTarget: String = null, extraTags: Map[TagDefinition, String], isSynchronous: Boolean = true, alwaysRecordStats: Boolean = false, allowAuthTags: Boolean = false, killJvmIfStuck: Boolean = false, outputMetric: MetricDefinition = METRIC_OPERATION_DURATION, silent: Boolean = true)(thunk: => S): S
- Definition Classes
- DatabricksLogging
- def recordProductEvent(metric: MetricDefinition with CentralizableMetric, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit
- Definition Classes
- DatabricksLogging
- def recordProductUsage(metric: MetricDefinition with CentralizableMetric, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit
- Definition Classes
- DatabricksLogging
- def recordUsage(metric: MetricDefinition, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit
- Definition Classes
- DatabricksLogging
- def removeClusteringColumnsProperty(configuration: Map[String, String]): Map[String, String]
Remove PROP_CLUSTERING_COLUMNS from metadata action.
Remove PROP_CLUSTERING_COLUMNS from metadata action. Clustering columns should only exist in: 1. CatalogTable.properties(PROP_CLUSTERING_COLUMNS) 2. Clustering metadata domain.
- configuration
original configuration.
- returns
new configuration without clustering columns property
- Definition Classes
- ClusteredTableUtilsBase
- def removeInternalTableProperties(props: Map[String, String]): Map[String, String]
Remove clustered table internal table properties.
Remove clustered table internal table properties. These properties are never stored into Metadata.configuration such as table features.
- Definition Classes
- ClusteredTableUtilsBase
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- def validateClusteringColumnsInSnapshot(snapshot: Snapshot, clusterBySpec: ClusterBySpec): Unit
Validate that the given clusterBySpec matches the existing table's in the given snapshot.
Validate that the given clusterBySpec matches the existing table's in the given snapshot. This is used for append mode and replaceWhere.
- Definition Classes
- ClusteredTableUtilsBase
- def validateClusteringColumnsInStatsSchema(protocol: Protocol, metadata: Metadata, clusterBy: ClusterBySpec): Unit
Validate stats will be collected for all clustering columns.
Validate stats will be collected for all clustering columns.
This version is used when Snapshot doesn't have latest stats column information such as
CREATE TABLE...where the initial snapshot doesn't have updated metadata / protocol yet.- Definition Classes
- ClusteredTableUtilsBase
- def validateClusteringColumnsInStatsSchema(snapshot: Snapshot, logicalClusteringColumns: Seq[String]): Unit
Validate stats will be collected for all clustering columns.
Validate stats will be collected for all clustering columns.
- Definition Classes
- ClusteredTableUtilsBase
- def validateExistingTableFeatureProperties(existingProperties: Map[String, String]): Unit
Verify user didn't set clustering table feature in table properties.
Verify user didn't set clustering table feature in table properties.
- existingProperties
Table properties set by the user when creating a clustered table.
- Definition Classes
- ClusteredTableUtilsBase
- def validateNumClusteringColumns(clusteringColumns: Seq[Seq[String]], deltaLogOpt: Option[DeltaLog] = None): Unit
Validate the number of clustering columns doesn't exceed the limit.
Validate the number of clustering columns doesn't exceed the limit.
- clusteringColumns
clustering columns for the table.
- deltaLogOpt
optional delta log. If present, will be used to record a delta event.
- Definition Classes
- ClusteredTableUtilsBase
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- def withStatusCode[T](statusCode: String, defaultMessage: String, data: Map[String, Any] = Map.empty)(body: => T): T
Report a log to indicate some command is running.
Report a log to indicate some command is running.
- Definition Classes
- DeltaProgressReporter