case class DeltaSink(sqlContext: SQLContext, path: Path, partitionColumns: Seq[String], outputMode: OutputMode, options: DeltaOptions, catalogTable: Option[CatalogTable] = None) extends Sink with ImplicitMetadataOperation with UpdateExpressionsSupport with DeltaLogging with Product with Serializable

A streaming sink that writes data into a Delta Table.

Linear Supertypes
Serializable, Serializable, Product, Equals, UpdateExpressionsSupport, AnalysisHelper, SQLConfHelper, ImplicitMetadataOperation, DeltaLogging, DatabricksLogging, DeltaProgressReporter, LoggingShims, Logging, Sink, Table, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DeltaSink
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. UpdateExpressionsSupport
  7. AnalysisHelper
  8. SQLConfHelper
  9. ImplicitMetadataOperation
  10. DeltaLogging
  11. DatabricksLogging
  12. DeltaProgressReporter
  13. LoggingShims
  14. Logging
  15. Sink
  16. Table
  17. AnyRef
  18. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new DeltaSink(sqlContext: SQLContext, path: Path, partitionColumns: Seq[String], outputMode: OutputMode, options: DeltaOptions, catalogTable: Option[CatalogTable] = None)

Type Members

  1. implicit class LogStringContext extends AnyRef
    Definition Classes
    LoggingShims
  2. case class UpdateOperation(targetColNameParts: Seq[String], updateExpr: Expression) extends Product with Serializable

    Specifies an operation that updates a target column with the given expression.

    Specifies an operation that updates a target column with the given expression. The target column may or may not be a nested field and it is specified as a full quoted name or as a sequence of split into parts.

    Definition Classes
    UpdateExpressionsSupport
  3. case class PendingTxn(batchId: Long, optimisticTransaction: OptimisticTransaction, streamingUpdate: StreamingUpdate, newFiles: Seq[FileAction], deletedFiles: Seq[FileAction]) extends Product with Serializable

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def addBatch(batchId: Long, data: DataFrame): Unit
    Definition Classes
    DeltaSink → Sink
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. val canMergeSchema: Boolean
    Attributes
    protected
    Definition Classes
    DeltaSink → ImplicitMetadataOperation
  7. val canOverwriteSchema: Boolean
    Attributes
    protected
    Definition Classes
    DeltaSink → ImplicitMetadataOperation
  8. def capabilities(): Set[TableCapability]
    Definition Classes
    Sink → Table
  9. def castIfNeeded(fromExpression: Expression, dataType: DataType, allowStructEvolution: Boolean, columnName: String): Expression

    Add a cast to the child expression if it differs from the specified data type.

    Add a cast to the child expression if it differs from the specified data type. Note that structs here are cast by name, rather than the Spark SQL default of casting by position.

    fromExpression

    the expression to cast

    dataType

    The data type to cast to.

    allowStructEvolution

    Whether to allow structs to evolve. When this is false (default), struct casting will throw an error if the target struct type contains more fields than the expression to cast.

    columnName

    The name of the column written to. It is used for the error message.

    Attributes
    protected
    Definition Classes
    UpdateExpressionsSupport
  10. val catalogTable: Option[CatalogTable]
  11. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  12. def columns(): Array[connector.catalog.Column]
    Definition Classes
    Table
  13. def conf: SQLConf
    Definition Classes
    SQLConfHelper
  14. def deltaAssert(check: ⇒ Boolean, name: String, msg: String, deltaLog: DeltaLog = null, data: AnyRef = null, path: Option[Path] = None): Unit

    Helper method to check invariants in Delta code.

    Helper method to check invariants in Delta code. Fails when running in tests, records a delta assertion event and logs a warning otherwise.

    Attributes
    protected
    Definition Classes
    DeltaLogging
  15. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  16. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  17. def generateUpdateExpressions(targetSchema: StructType, defaultExprs: Seq[NamedExpression], nameParts: Seq[Seq[String]], updateExprs: Seq[Expression], resolver: Resolver, generatedColumns: Seq[StructField]): Seq[Option[Expression]]

    See docs on overloaded method.

    See docs on overloaded method.

    Attributes
    protected
    Definition Classes
    UpdateExpressionsSupport
  18. def generateUpdateExpressions(targetSchema: StructType, updateOps: Seq[UpdateOperation], defaultExprs: Seq[NamedExpression], resolver: Resolver, pathPrefix: Seq[String] = Nil, allowSchemaEvolution: Boolean = false, generatedColumns: Seq[StructField] = Nil): Seq[Option[Expression]]

    Given a target schema and a set of update operations, generate a list of update expressions, which are aligned with the given schema.

    Given a target schema and a set of update operations, generate a list of update expressions, which are aligned with the given schema.

    For update operations to nested struct fields, this method recursively walks down schema tree and apply the update expressions along the way. For example, assume table target has the following schema: s1 struct<a: int, b: int, c: int>, s2 struct<a: int, b: int>, z int

    Given an update command:

    • UPDATE target SET s1.a = 1, s1.b = 2, z = 3

    this method works as follows:

    generateUpdateExpressions( targetSchema=[s1,s2,z], defaultExprs=[s1,s2, z], updateOps=[(s1.a, 1), (s1.b, 2), (z, 3)]) -> generates expression for s1 - build recursively from child assignments generateUpdateExpressions( targetSchema=[a,b,c], defaultExprs=[a, b, c], updateOps=[(a, 1),(b, 2)], pathPrefix=["s1"]) end-of-recursion -> returns (1, 2, a.c) -> generates expression for s2 - no child assignment and no update expression: use default expression s2 -> generates expression for z - use available update expression 3 -> returns ((1, 2, a.c), s2, 3)

    targetSchema

    schema to follow to generate update expressions. Due to schema evolution, it may contain additional columns or fields not present in the original table schema.

    updateOps

    a set of update operations.

    defaultExprs

    the expressions to use when no update operation is provided for a column or field. This is typically the output from the base table.

    pathPrefix

    the path from root to the current (nested) column. Only used for printing out full column path in error messages.

    allowSchemaEvolution

    Whether to allow generating expressions for new columns or fields added by schema evolution.

    generatedColumns

    the list of the generated columns in the table. When a column is a generated column and the user doesn't provide a update expression, its update expression in the return result will be None. If generatedColumns is empty, any of the options in the return result must be non-empty.

    returns

    a sequence of expression options. The elements in the sequence are options because when a column is a generated column but the user doesn't provide an update expression for this column, we need to generate the update expression according to the generated column definition. But this method doesn't have enough context to do that. Hence, we return a None for this case so that the caller knows it should generate the update expression for such column. For other cases, we will always return Some(expr).

    Attributes
    protected
    Definition Classes
    UpdateExpressionsSupport
  19. def generateUpdateExprsForGeneratedColumns(updateTarget: LogicalPlan, generatedColumns: Seq[StructField], updateExprs: Seq[Option[Expression]], postEvolutionTargetSchema: Option[StructType] = None): Seq[Expression]

    Generate update expressions for generated columns that the user doesn't provide a update expression.

    Generate update expressions for generated columns that the user doesn't provide a update expression. For each item in updateExprs that's None, we will find its generation expression from generatedColumns. In order to resolve this generation expression, we will create a fake Project which contains all update expressions and resolve the generation expression with this project. Source columns of a generation expression will also be replaced with their corresponding update expressions.

    For example, given a table that has a generated column g defined as c1 + 10. For the following update command:

    UPDATE target SET c1 = c2 + 100, c2 = 1000

    We will generate the update expression (c2 + 100) + 10 for column g. Note: in this update expression, we should use the old c2 attribute rather than its new value 1000.

    updateTarget

    The logical plan of the table to be updated.

    generatedColumns

    A list of generated columns.

    updateExprs

    The aligned (with postEvolutionTargetSchema if not None, or updateTarget.output otherwise) update actions.

    postEvolutionTargetSchema

    In case of UPDATE in MERGE when schema evolution happened, this is the final schema of the target table. This might not be the same as the output of updateTarget.

    returns

    a sequence of update expressions for all of columns in the table.

    Attributes
    protected
    Definition Classes
    UpdateExpressionsSupport
  20. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  21. def getCommonTags(deltaLog: DeltaLog, tahoeId: String): Map[TagDefinition, String]
    Definition Classes
    DeltaLogging
  22. def getErrorData(e: Throwable): Map[String, Any]
    Definition Classes
    DeltaLogging
  23. final def getNewDomainMetadata(txn: OptimisticTransaction, canUpdateMetadata: Boolean, isReplacingTable: Boolean, clusterBySpecOpt: Option[ClusterBySpec] = None): Seq[DomainMetadata]

    Returns a sequence of new DomainMetadata if canUpdateMetadata is true and the operation is either create table or replace the whole table (not replaceWhere operation).

    Returns a sequence of new DomainMetadata if canUpdateMetadata is true and the operation is either create table or replace the whole table (not replaceWhere operation). This is because we only update Domain Metadata when creating or replacing table, and replace table for DDL and DataFrameWriterV2 are already handled in CreateDeltaTableCommand. In that case, canUpdateMetadata is false, so we don't update again.

    txn

    OptimisticTransaction being used to create or replace table.

    canUpdateMetadata

    true if the metadata is not updated yet.

    isReplacingTable

    true if the operation is replace table without replaceWhere option.

    clusterBySpecOpt

    optional ClusterBySpec containing user-specified clustering columns.

    Attributes
    protected
    Definition Classes
    ImplicitMetadataOperation
  24. def improveUnsupportedOpError(f: ⇒ Unit): Unit
    Attributes
    protected
    Definition Classes
    AnalysisHelper
  25. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  26. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  27. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  28. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  29. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  30. def logConsole(line: String): Unit
    Definition Classes
    DatabricksLogging
  31. def logDebug(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    LoggingShims
  32. def logDebug(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    LoggingShims
  33. def logDebug(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  34. def logDebug(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  35. def logError(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    LoggingShims
  36. def logError(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    LoggingShims
  37. def logError(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  38. def logError(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  39. def logInfo(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    LoggingShims
  40. def logInfo(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    LoggingShims
  41. def logInfo(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  42. def logInfo(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  43. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  44. def logTrace(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    LoggingShims
  45. def logTrace(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    LoggingShims
  46. def logTrace(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  47. def logTrace(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  48. def logWarning(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    LoggingShims
  49. def logWarning(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    LoggingShims
  50. def logWarning(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  51. def logWarning(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  52. def name(): String
    Definition Classes
    Sink → Table
  53. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  54. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  55. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  56. val options: DeltaOptions
  57. val outputMode: OutputMode
  58. val partitionColumns: Seq[String]
  59. def partitioning(): Array[Transform]
    Definition Classes
    Table
  60. val path: Path
  61. def properties(): Map[String, String]
    Definition Classes
    Table
  62. lazy val queryId: String
  63. def recordDeltaEvent(deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty, data: AnyRef = null, path: Option[Path] = None): Unit

    Used to record the occurrence of a single event or report detailed, operation specific statistics.

    Used to record the occurrence of a single event or report detailed, operation specific statistics.

    path

    Used to log the path of the delta table when deltaLog is null.

    Attributes
    protected
    Definition Classes
    DeltaLogging
  64. def recordDeltaOperation[A](deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: ⇒ A): A

    Used to report the duration as well as the success or failure of an operation on a deltaLog.

    Used to report the duration as well as the success or failure of an operation on a deltaLog.

    Attributes
    protected
    Definition Classes
    DeltaLogging
  65. def recordDeltaOperationForTablePath[A](tablePath: String, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: ⇒ A): A

    Used to report the duration as well as the success or failure of an operation on a tahoePath.

    Used to report the duration as well as the success or failure of an operation on a tahoePath.

    Attributes
    protected
    Definition Classes
    DeltaLogging
  66. def recordEvent(metric: MetricDefinition, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit
    Definition Classes
    DatabricksLogging
  67. def recordFrameProfile[T](group: String, name: String)(thunk: ⇒ T): T
    Attributes
    protected
    Definition Classes
    DeltaLogging
  68. def recordOperation[S](opType: OpType, opTarget: String = null, extraTags: Map[TagDefinition, String], isSynchronous: Boolean = true, alwaysRecordStats: Boolean = false, allowAuthTags: Boolean = false, killJvmIfStuck: Boolean = false, outputMetric: MetricDefinition = METRIC_OPERATION_DURATION, silent: Boolean = true)(thunk: ⇒ S): S
    Definition Classes
    DatabricksLogging
  69. def recordProductEvent(metric: MetricDefinition with CentralizableMetric, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit
    Definition Classes
    DatabricksLogging
  70. def recordProductUsage(metric: MetricDefinition with CentralizableMetric, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit
    Definition Classes
    DatabricksLogging
  71. def recordUsage(metric: MetricDefinition, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit
    Definition Classes
    DatabricksLogging
  72. def resolveReferencesForExpressions(sparkSession: SparkSession, exprs: Seq[Expression], planProvidingAttrs: LogicalPlan): Seq[Expression]

    Resolve expressions using the attributes provided by planProvidingAttrs.

    Resolve expressions using the attributes provided by planProvidingAttrs. Throw an error if failing to resolve any expressions.

    Attributes
    protected
    Definition Classes
    AnalysisHelper
  73. def schema(): StructType
    Definition Classes
    Sink → Table
  74. val sqlContext: SQLContext
  75. val supportMergeAndUpdateLegacyCastBehavior: Boolean

    Whether casting behavior can revert to following 'spark.sql.ansi.enabled' instead of 'spark.sql.storeAssignmentPolicy' to preserve legacy behavior for UPDATE and MERGE.

    Whether casting behavior can revert to following 'spark.sql.ansi.enabled' instead of 'spark.sql.storeAssignmentPolicy' to preserve legacy behavior for UPDATE and MERGE. Legacy behavior is applied only if 'spark.databricks.delta.updateAndMergeCastingFollowsAnsiEnabledFlag' is set to true.

    Attributes
    protected
    Definition Classes
    UpdateExpressionsSupport
  76. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  77. def toDataset(sparkSession: SparkSession, logicalPlan: LogicalPlan): Dataset[Row]
    Attributes
    protected
    Definition Classes
    AnalysisHelper
  78. def toString(): String
    Definition Classes
    DeltaSink → AnyRef → Any
  79. def tryResolveReferences(sparkSession: SparkSession)(expr: Expression, planContainingExpr: LogicalPlan): Expression
    Attributes
    protected
    Definition Classes
    AnalysisHelper
  80. def tryResolveReferencesForExpressions(sparkSession: SparkSession)(exprs: Seq[Expression], plansProvidingAttrs: Seq[LogicalPlan]): Seq[Expression]

    Resolve expressions using the attributes provided by planProvidingAttrs, ignoring errors.

    Resolve expressions using the attributes provided by planProvidingAttrs, ignoring errors.

    Attributes
    protected
    Definition Classes
    AnalysisHelper
  81. def tryResolveReferencesForExpressions(sparkSession: SparkSession, exprs: Seq[Expression], planContainingExpr: LogicalPlan): Seq[Expression]
    Attributes
    protected
    Definition Classes
    AnalysisHelper
  82. final def updateMetadata(spark: SparkSession, txn: OptimisticTransaction, schema: StructType, partitionColumns: Seq[String], configuration: Map[String, String], isOverwriteMode: Boolean, rearrangeOnly: Boolean): Unit
    Attributes
    protected
    Definition Classes
    ImplicitMetadataOperation
  83. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  84. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  85. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  86. def withStatusCode[T](statusCode: String, defaultMessage: String, data: Map[String, Any] = Map.empty)(body: ⇒ T): T

    Report a log to indicate some command is running.

    Report a log to indicate some command is running.

    Definition Classes
    DeltaProgressReporter

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from UpdateExpressionsSupport

Inherited from AnalysisHelper

Inherited from SQLConfHelper

Inherited from ImplicitMetadataOperation

Inherited from DeltaLogging

Inherited from DatabricksLogging

Inherited from DeltaProgressReporter

Inherited from LoggingShims

Inherited from Logging

Inherited from Sink

Inherited from Table

Inherited from AnyRef

Inherited from Any

Ungrouped