Value Members

1.  final def !=(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

2.  final def ##: Int

   Definition Classes

   AnyRef → Any

3.  final def ==(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

4.  val COLUMN_MAPPING_METADATA_ID_KEY: String
5.  val COLUMN_MAPPING_METADATA_NESTED_IDS_KEY: String
6.  val COLUMN_MAPPING_METADATA_PREFIX: String
7.  val COLUMN_MAPPING_PHYSICAL_NAME_KEY: String
8.  val DELTA_INTERNAL_COLUMNS: Set[String]

   This list of internal columns (and only this list) is allowed to have missing column mapping metadata such as field id and physical name because they might not be present in user's table schema.

   This list of internal columns (and only this list) is allowed to have missing column mapping metadata such as field id and physical name because they might not be present in user's table schema.

   These fields, if materialized to parquet, will always be matched by their display name in the downstream parquet reader even under column mapping modes.

   For future developers who want to utilize additional internal columns without generating column mapping metadata, please add them here.

   This list is case-insensitive.

   Attributes

   protected

9.  val PARQUET_FIELD_ID_METADATA_KEY: String
10.  val PARQUET_FIELD_NESTED_IDS_METADATA_KEY: String
11.  val PARQUET_LIST_ELEMENT_FIELD_NAME: String
12.  val PARQUET_MAP_KEY_FIELD_NAME: String
13.  val PARQUET_MAP_VALUE_FIELD_NAME: String
14.  final def asInstanceOf[T0]: T0

    Definition Classes

    Any

15.  def assignColumnIdAndPhysicalName(newMetadata: Metadata, oldMetadata: Metadata, isChangingModeOnExistingTable: Boolean, isOverwritingSchema: Boolean): Metadata

    For each column/field in a Metadata's schema, assign id using the current maximum id as the basis and increment from there, and assign physical name using UUID

    For each column/field in a Metadata's schema, assign id using the current maximum id as the basis and increment from there, and assign physical name using UUID

    newMetadata

    The new metadata to assign Ids and physical names

    oldMetadata

    The old metadata

    isChangingModeOnExistingTable

    whether this is part of a commit that changes the mapping mode on a existing table

    returns

    new metadata with Ids and physical names assigned

16.  def assignPhysicalName(field: StructField, physicalName: String): StructField
17.  def assignPhysicalNames(schema: StructType): StructType
18.  def checkColumnIdAndPhysicalNameAssignments(metadata: Metadata): Unit

    Verify the metadata for valid column mapping metadata assignment.

    Verify the metadata for valid column mapping metadata assignment. This is triggered for every commit as a last defense.

    1. Ensure column mapping metadata is set for the appropriate mode 2. Ensure no duplicate column id/physical names set 3. Ensure max column id is in a good state (set, and greater than all field ids available)

19.  def clone(): AnyRef

    Attributes

    protected[lang]

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.CloneNotSupportedException]) @native()

20.  def createPhysicalAttributes(output: Seq[Attribute], referenceSchema: StructType, columnMappingMode: DeltaColumnMappingMode): Seq[Attribute]

    Create a list of physical attributes for the given attributes using the table schema as a reference.

    Create a list of physical attributes for the given attributes using the table schema as a reference.

    output

    the list of attributes (potentially without any metadata)

    referenceSchema

    the table schema with all the metadata

    columnMappingMode

    column mapping mode of the delta table, which determines which metadata to fill in

21.  def createPhysicalSchema(schema: StructType, referenceSchema: StructType, columnMappingMode: DeltaColumnMappingMode, checkSupportedMode: Boolean = true): StructType

    Create a physical schema for the given schema using the Delta table schema as a reference.

    Create a physical schema for the given schema using the Delta table schema as a reference.

    schema

    the given logical schema (potentially without any metadata)

    referenceSchema

    the schema from the delta log, which has all the metadata

    columnMappingMode

    column mapping mode of the delta table, which determines which metadata to fill in

    checkSupportedMode

    whether we should check of the column mapping mode is supported

22.  def deltaAssert(check: => Boolean, name: String, msg: String, deltaLog: DeltaLog = null, data: AnyRef = null, path: Option[Path] = None): Unit

    Helper method to check invariants in Delta code.

    Helper method to check invariants in Delta code. Fails when running in tests, records a delta assertion event and logs a warning otherwise.

    Attributes

    protected

    Definition Classes

    DeltaLogging

23.  def dropColumnMappingMetadata(schema: StructType): StructType
24.  final def eq(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

25.  def equals(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef → Any

26.  def filterColumnMappingProperties(properties: Map[String, String]): Map[String, String]
27.  def finalize(): Unit

    Attributes

    protected[lang]

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.Throwable])

28.  def findMaxColumnId(schema: StructType): Long
29.  def generatePhysicalName: String
30.  final def getClass(): Class[_ <: AnyRef]

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

31.  def getColumnId(field: StructField): Int
32.  def getColumnMappingMetadata(field: StructField, mode: DeltaColumnMappingMode): Metadata

    Gets the required column metadata for each column based on the column mapping mode.

33.  def getCommonTags(deltaLog: DeltaLog, tahoeId: String): Map[TagDefinition, String]

    Definition Classes

    DeltaLogging

34.  def getErrorData(e: Throwable): Map[String, Any]

    Definition Classes

    DeltaLogging

35.  def getLogicalNameToPhysicalNameMap(schema: StructType): Map[Seq[String], Seq[String]]

    Returns a map from the logical name paths to the physical name paths for the given schema.

    Returns a map from the logical name paths to the physical name paths for the given schema. The logical name path is the result of splitting a multi-part identifier, and the physical name path is result of replacing all names in the logical name path with their physical names.

36.  def getNestedColumnIds(field: StructField): Metadata
37.  def getPhysicalName(field: StructField): String
38.  def getPhysicalNameFieldMap(schema: StructType): Map[Seq[String], StructField]

    Returns a map of physicalNamePath -> field for the given schema, where physicalNamePath is the [$parentPhysicalName, ..., $fieldPhysicalName] list of physical names for every field (including nested) in the schema.

    Returns a map of physicalNamePath -> field for the given schema, where physicalNamePath is the [$parentPhysicalName, ..., $fieldPhysicalName] list of physical names for every field (including nested) in the schema.

    Must be called after checkColumnIdAndPhysicalNameAssignments, so that we know the schema is valid.

39.  def hasColumnId(field: StructField): Boolean
40.  def hasNestedColumnIds(field: StructField): Boolean
41.  def hasNoColumnMappingSchemaChanges(newMetadata: Metadata, oldMetadata: Metadata, allowUnsafeReadOnPartitionChanges: Boolean = false): Boolean

    Compare the old metadata's schema with new metadata's schema for column mapping schema changes.

    Compare the old metadata's schema with new metadata's schema for column mapping schema changes. Also check for repartition because we need to fail fast when repartition detected.

    newMetadata's snapshot version must be >= oldMetadata's snapshot version so we could reliably detect the difference between ADD COLUMN and DROP COLUMN.

    As of now, newMetadata is column mapping read compatible with oldMetadata if no rename column or drop column has happened in-between.

42.  def hasPhysicalName(field: StructField): Boolean
43.  def hashCode(): Int

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

44.  def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

    Attributes

    protected

    Definition Classes

    Logging

45.  def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Attributes

    protected

    Definition Classes

    Logging

46.  def isColumnMappingUpgrade(oldMode: DeltaColumnMappingMode, newMode: DeltaColumnMappingMode): Boolean
47.  def isDropColumnOperation(newSchema: StructType, currentSchema: StructType): Boolean
48.  def isDropColumnOperation(newMetadata: Metadata, currentMetadata: Metadata): Boolean

    Returns true if Column Mapping mode is enabled and the newMetadata's schema, when compared to the currentMetadata's schema, is indicative of a DROP COLUMN operation.

    Returns true if Column Mapping mode is enabled and the newMetadata's schema, when compared to the currentMetadata's schema, is indicative of a DROP COLUMN operation.

    We detect DROP COLUMNS by checking if any physical name in currentSchema is missing in newSchema.

49.  final def isInstanceOf[T0]: Boolean

    Definition Classes

    Any

50.  def isInternalField(field: StructField): Boolean
51.  def isRenameColumnOperation(newSchema: StructType, currentSchema: StructType): Boolean
52.  def isRenameColumnOperation(newMetadata: Metadata, currentMetadata: Metadata): Boolean

    Returns true if Column Mapping mode is enabled and the newMetadata's schema, when compared to the currentMetadata's schema, is indicative of a RENAME COLUMN operation.

    Returns true if Column Mapping mode is enabled and the newMetadata's schema, when compared to the currentMetadata's schema, is indicative of a RENAME COLUMN operation.

    We detect RENAME COLUMNS by checking if any two columns with the same physical name have different logical names

53.  def isTraceEnabled(): Boolean

    Attributes

    protected

    Definition Classes

    Logging

54.  def log: Logger

    Attributes

    protected

    Definition Classes

    Logging

55.  def logConsole(line: String): Unit

    Definition Classes

    DatabricksLogging

56.  def logDebug(msg: => String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

57.  def logDebug(msg: => String): Unit

    Attributes

    protected

    Definition Classes

    Logging

58.  def logError(msg: => String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

59.  def logError(msg: => String): Unit

    Attributes

    protected

    Definition Classes

    Logging

60.  def logInfo(msg: => String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

61.  def logInfo(msg: => String): Unit

    Attributes

    protected

    Definition Classes

    Logging

62.  def logName: String

    Attributes

    protected

    Definition Classes

    Logging

63.  def logTrace(msg: => String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

64.  def logTrace(msg: => String): Unit

    Attributes

    protected

    Definition Classes

    Logging

65.  def logWarning(msg: => String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

66.  def logWarning(msg: => String): Unit

    Attributes

    protected

    Definition Classes

    Logging

67.  final def ne(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

68.  final def notify(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

69.  final def notifyAll(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

70.  def recordDeltaEvent(deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty, data: AnyRef = null, path: Option[Path] = None): Unit

    Used to record the occurrence of a single event or report detailed, operation specific statistics.

    Used to record the occurrence of a single event or report detailed, operation specific statistics.

    path

    Used to log the path of the delta table when deltaLog is null.

    Attributes

    protected

    Definition Classes

    DeltaLogging

71.  def recordDeltaOperation[A](deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: => A): A

    Used to report the duration as well as the success or failure of an operation on a deltaLog.

    Used to report the duration as well as the success or failure of an operation on a deltaLog.

    Attributes

    protected

    Definition Classes

    DeltaLogging

72.  def recordDeltaOperationForTablePath[A](tablePath: String, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: => A): A

    Used to report the duration as well as the success or failure of an operation on a tahoePath.

    Used to report the duration as well as the success or failure of an operation on a tahoePath.

    Attributes

    protected

    Definition Classes

    DeltaLogging

73.  def recordEvent(metric: MetricDefinition, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit

    Definition Classes

    DatabricksLogging

74.  def recordFrameProfile[T](group: String, name: String)(thunk: => T): T

    Attributes

    protected

    Definition Classes

    DeltaLogging

75.  def recordOperation[S](opType: OpType, opTarget: String = null, extraTags: Map[TagDefinition, String], isSynchronous: Boolean = true, alwaysRecordStats: Boolean = false, allowAuthTags: Boolean = false, killJvmIfStuck: Boolean = false, outputMetric: MetricDefinition = METRIC_OPERATION_DURATION, silent: Boolean = true)(thunk: => S): S

    Definition Classes

    DatabricksLogging

76.  def recordProductEvent(metric: MetricDefinition with CentralizableMetric, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit

    Definition Classes

    DatabricksLogging

77.  def recordProductUsage(metric: MetricDefinition with CentralizableMetric, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit

    Definition Classes

    DatabricksLogging

78.  def recordUsage(metric: MetricDefinition, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit

    Definition Classes

    DatabricksLogging

79.  def renameColumns(schema: StructType): StructType

    Recursively renames columns in the given schema with their physical schema.

80.  def rewriteFieldIdsForIceberg(schema: StructType, startId: Long): (StructType, Long)

    Adds the nested field IDs required by Iceberg.

    Adds the nested field IDs required by Iceberg.

    In parquet, list-type columns have a nested, implicitly defined element field and map-type columns have implicitly defined key and value fields. By default, Spark does not write field IDs for these fields in the parquet files. However, Iceberg requires these *nested* field IDs to be present. This method rewrites the specified Spark schema to add those nested field IDs.

    As list and map types are not StructFields themselves, nested field IDs are stored in a map as part of the metadata of the *nearest* parent StructField. For example, consider the following schema:

    col1 ARRAY(INT) col2 MAP(INT, INT) col3 STRUCT(a INT, b ARRAY(STRUCT(c INT, d MAP(INT, INT))))

    col1 is a list and so requires one nested field ID for the element field in parquet. This nested field ID will be stored in a map that is part of col1's StructField.metadata. The same applies to the nested field IDs for col2's implicit key and value fields. col3 itself is a Struct, consisting of an integer field and a list field named 'b'. The nested field ID for the list of 'b' is stored in b's StructField metadata. Finally, the list type itself is again a struct consisting of an integer field and a map field named 'd'. The nested field IDs for the map of 'd' are stored in d's StructField metadata.

    schema

    The schema to which nested field IDs should be added

    startId

    The first field ID to use for the nested field IDs

81.  def satisfiesColumnMappingProtocol(protocol: Protocol): Boolean
82.  def setPhysicalNames(schema: StructType, fieldPathToPhysicalName: Map[Seq[String], String]): StructType

    Set physical name based on field path, skip if field path not found in the map

83.  val supportedModes: Set[DeltaColumnMappingMode]
84.  final def synchronized[T0](arg0: => T0): T0

    Definition Classes

    AnyRef

85.  def toString(): String

    Definition Classes

    AnyRef → Any

86.  def verifyAndUpdateMetadataChange(deltaLog: DeltaLog, oldProtocol: Protocol, oldMetadata: Metadata, newMetadata: Metadata, isCreatingNewTable: Boolean, isOverwriteSchema: Boolean): Metadata

    If the table is already on the column mapping protocol, we block:

    If the table is already on the column mapping protocol, we block:

    • changing column mapping config otherwise, we block
    • upgrading to the column mapping Protocol through configurations
87.  def verifyInternalProperties(one: Map[String, String], two: Map[String, String]): Boolean
88.  final def wait(): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.InterruptedException])

89.  final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.InterruptedException])

90.  final def wait(arg0: Long): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws(classOf[java.lang.InterruptedException]) @native()

91.  def withStatusCode[T](statusCode: String, defaultMessage: String, data: Map[String, Any] = Map.empty)(body: => T): T

    Report a log to indicate some command is running.

    Report a log to indicate some command is running.

    Definition Classes

    DeltaProgressReporter