Packages

  • package root
    Definition Classes
    root
  • package org
    Definition Classes
    root
  • package apache
    Definition Classes
    org
  • package spark
    Definition Classes
    apache
  • package sql
    Definition Classes
    spark
  • package delta
    Definition Classes
    sql
  • package commands
    Definition Classes
    delta
  • package merge
    Definition Classes
    commands
  • trait MergeIntoMaterializeSource extends DeltaLogging with DeltaSparkPlanUtils

    Trait with logic and utilities used for materializing a snapshot of MERGE source in case we can't guarantee deterministic repeated reads from it.

    Trait with logic and utilities used for materializing a snapshot of MERGE source in case we can't guarantee deterministic repeated reads from it.

    We materialize source if it is not safe to assume that it's deterministic (override with MERGE_SOURCE_MATERIALIZATION). Otherwise, if source changes between the phases of the MERGE, it can produce wrong results. We use local checkpointing for the materialization, which saves the source as a materialized RDD[InternalRow] on the executor local disks.

    1st concern is that if an executor is lost, this data can be lost. When Spark executor decommissioning API is used, it should attempt to move this materialized data safely out before removing the executor.

    2nd concern is that if an executor is lost for another reason (e.g. spot kill), we will still lose that data. To mitigate that, we implement a retry loop. The whole Merge operation needs to be restarted from the beginning in this case. When we retry, we increase the replication level of the materialized data from 1 to 2. (override with MERGE_SOURCE_MATERIALIZATION_RDD_STORAGE_LEVEL_RETRY). If it still fails after the maximum number of attempts (MERGE_MATERIALIZE_SOURCE_MAX_ATTEMPTS), we record the failure for tracking purposes.

    3rd concern is that executors run out of disk space with the extra materialization. We record such failures for tracking purposes.

    Definition Classes
    merge
  • LogStringContext
  • RetryHandling
  • SubqueryExpression

object RetryHandling extends Enumeration

Linear Supertypes
Enumeration, Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. RetryHandling
  2. Enumeration
  3. Serializable
  4. Serializable
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. type Result = Value
  2. class Val extends Value with Serializable
    Attributes
    protected
    Definition Classes
    Enumeration
    Annotations
    @SerialVersionUID()
  3. abstract class Value extends Ordered[Value] with Serializable
    Definition Classes
    Enumeration
    Annotations
    @SerialVersionUID()
  4. class ValueSet extends AbstractSet[Value] with SortedSet[Value] with SortedSetLike[Value, ValueSet] with Serializable
    Definition Classes
    Enumeration

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. val ExhaustedRetries: Value
  5. val RethrowException: Value
  6. val Retry: Value
  7. final def Value(i: Int, name: String): Value
    Attributes
    protected
    Definition Classes
    Enumeration
  8. final def Value(name: String): Value
    Attributes
    protected
    Definition Classes
    Enumeration
  9. final def Value(i: Int): Value
    Attributes
    protected
    Definition Classes
    Enumeration
  10. final def Value: Value
    Attributes
    protected
    Definition Classes
    Enumeration
  11. final def apply(x: Int): Value
    Definition Classes
    Enumeration
  12. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  13. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  14. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  15. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  16. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  17. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  18. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  19. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  20. final def maxId: Int
    Definition Classes
    Enumeration
  21. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  22. var nextId: Int
    Attributes
    protected
    Definition Classes
    Enumeration
  23. var nextName: Iterator[String]
    Attributes
    protected
    Definition Classes
    Enumeration
  24. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  25. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  26. def readResolve(): AnyRef
    Attributes
    protected
    Definition Classes
    Enumeration
  27. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  28. def toString(): String
    Definition Classes
    Enumeration → AnyRef → Any
  29. def values: ValueSet
    Definition Classes
    Enumeration
  30. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  31. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  32. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  33. final def withName(s: String): Value
    Definition Classes
    Enumeration

Inherited from Enumeration

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped