Packages

o

org.apache.spark.sql.delta

DeltaOperations

object DeltaOperations

Exhaustive list of operations that can be performed on a Delta table. These operations are tracked as the first line in delta logs, and power DESCRIBE HISTORY for Delta tables.

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DeltaOperations
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. case class AddColumns(colsToAdd: Seq[QualifiedColTypeWithPositionForLog]) extends Operation with Product with Serializable

    Recorded when columns are added.

  2. case class AddConstraint(constraintName: String, expr: String) extends Operation with Product with Serializable
  3. case class ChangeColumn(columnPath: Seq[String], columnName: String, newColumn: StructField, colPosition: Option[String]) extends Operation with Product with Serializable

    Recorded when columns are changed.

  4. case class Clone(source: String, sourceVersion: Long) extends Operation with Product with Serializable
  5. case class ClusterBy(oldClusteringColumns: String, newClusteringColumns: String) extends Operation with Product with Serializable

    Recorded when clustering columns are changed on clustered tables.

  6. case class ComputeStats(predicate: Seq[Expression]) extends OperationWithPredicates with Product with Serializable

    Recorded when recomputing stats on the table.

  7. case class Convert(numFiles: Long, partitionBy: Seq[String], collectStats: Boolean, catalogTable: Option[String], sourceFormat: Option[String]) extends Operation with Product with Serializable

    Recorded when converting a table into a Delta table.

  8. case class CreateTable(metadata: Metadata, isManaged: Boolean, asSelect: Boolean = false, clusterBy: Option[Seq[String]] = None) extends Operation with Product with Serializable

    Recorded when the table is created.

  9. case class Delete(predicate: Seq[Expression]) extends OperationWithPredicates with Product with Serializable

    Recorded while deleting certain partitions.

  10. case class DropColumns(colsToDrop: Seq[Seq[String]]) extends Operation with Product with Serializable
  11. case class DropConstraint(constraintName: String, expr: Option[String]) extends Operation with Product with Serializable
  12. case class DropTableFeature(featureName: String, truncateHistory: Boolean) extends Operation with Product with Serializable

    Recorded when dropping a table feature.

  13. case class Merge(predicate: Option[Expression], updatePredicate: Option[String], deletePredicate: Option[String], insertPredicate: Option[String], matchedPredicates: Seq[MergePredicate], notMatchedPredicates: Seq[MergePredicate], notMatchedBySourcePredicates: Seq[MergePredicate]) extends OperationWithPredicates with Product with Serializable
  14. case class MergePredicate(predicate: Option[String], actionType: String) extends Product with Serializable

    Represents the predicates and action type (insert, update, delete) for a Merge clause

  15. sealed abstract class Operation extends AnyRef

    An operation that can be performed on a Delta table.

  16. abstract class OperationWithPredicates extends Operation
  17. case class Optimize(predicate: Seq[Expression], zOrderBy: Seq[String] = Seq.empty, auto: Boolean = false, clusterBy: Option[Seq[String]] = None, isFull: Boolean = false) extends OptimizeOrReorg with Product with Serializable

    Recorded when optimizing the table.

  18. sealed abstract class OptimizeOrReorg extends OperationWithPredicates
  19. case class QualifiedColTypeWithPositionForLog(columnPath: Seq[String], column: StructField, colPosition: Option[String]) extends Product with Serializable

    Qualified column type with position.

    Qualified column type with position. We define a copy of the type here to avoid depending on the parser output classes in our logging.

  20. case class RemoveColumnMapping(userMetadata: Option[String] = None) extends Operation with Product with Serializable
  21. case class RenameColumn(oldColumnPath: Seq[String], newColumnPath: Seq[String]) extends Operation with Product with Serializable
  22. case class Reorg(predicate: Seq[Expression], applyPurge: Boolean = true) extends OptimizeOrReorg with Product with Serializable

    Recorded when running REORG on the table.

  23. case class ReplaceColumns(columns: Seq[StructField]) extends Operation with Product with Serializable

    Recorded when columns are replaced.

  24. case class ReplaceTable(metadata: Metadata, isManaged: Boolean, orCreate: Boolean, asSelect: Boolean = false, userMetadata: Option[String] = None, clusterBy: Option[Seq[String]] = None) extends Operation with Product with Serializable

    Recorded when the table is replaced.

  25. case class Restore(version: Option[Long], timestamp: Option[String]) extends Operation with Product with Serializable
  26. case class RowTrackingBackfill(batchId: Int = 0) extends Operation with Product with Serializable

    Recorded when we backfill a Delta table's existing AddFiles with row tracking data.

  27. case class SetTableProperties(properties: Map[String, String]) extends Operation with Product with Serializable
  28. case class StreamingUpdate(outputMode: OutputMode, queryId: String, epochId: Long, userMetadata: Option[String] = None) extends Operation with Product with Serializable

    Recorded during streaming inserts.

  29. case class TestOperation(operationName: String = "TEST", isInPlaceFileMetadataUpdate: Option[Boolean] = None) extends Operation with Product with Serializable

    Dummy operation only for testing with arbitrary operation names

  30. case class Truncate() extends Operation with Product with Serializable

    Recorded when truncating the table.

  31. case class UnsetTableProperties(propKeys: Seq[String], ifExists: Boolean) extends Operation with Product with Serializable

    Recorded when the table properties are unset.

  32. case class Update(predicate: Option[Expression]) extends OperationWithPredicates with Product with Serializable

    Recorded when an update operation is committed to the table.

  33. case class UpdateColumnMetadata(operationName: String, columns: Seq[(Seq[String], StructField)]) extends Operation with Product with Serializable
  34. case class UpdateSchema(oldSchema: StructType, newSchema: StructType) extends Operation with Product with Serializable
  35. case class UpgradeProtocol(newProtocol: Protocol) extends Operation with Product with Serializable
  36. case class UpgradeUniformProperties(properties: Map[String, String]) extends Operation with Product with Serializable

    recorded when upgrading a table set uniform properties by REORG TABLE ...

    recorded when upgrading a table set uniform properties by REORG TABLE ... UPGRADE UNIFORM

  37. case class VacuumEnd(status: String) extends Operation with Product with Serializable

    status

    - whether the vacuum operation was successful; either "COMPLETED" or "FAILED"

  38. case class VacuumStart(retentionCheckEnabled: Boolean, specifiedRetentionMillis: Option[Long], defaultRetentionMillis: Long) extends Operation with Product with Serializable

    retentionCheckEnabled

    - whether retention check was enabled for this run of vacuum.

    specifiedRetentionMillis

    - specified retention interval

    defaultRetentionMillis

    - default retention period for the table

  39. case class Write(mode: SaveMode, partitionBy: Option[Seq[String]] = None, predicate: Option[String] = None, userMetadata: Option[String] = None) extends Operation with Product with Serializable

    Recorded during batch inserts.

    Recorded during batch inserts. Predicates can be provided for overwrites.

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. val AUTO_COMPACTION_PARAMETER_KEY: String

    parameter key to indicate whether it's an Auto Compaction

  5. val CLUSTERING_IS_FULL_KEY: String

    parameter key to indicate the operation for OPTIMIZE tbl FULL

  6. val CLUSTERING_PARAMETER_KEY: String

    parameter key to indicate clustering columns

  7. val OPTIMIZE_OPERATION_NAME: String

    operation name for OPTIMIZE command

  8. val OP_CLONE: String

    Recorded when cloning a Delta table into a new location.

  9. val OP_DROP_COLUMN: String

    Recorded when columns are dropped.

  10. val OP_MERGE: String

    Recorded when a merge operation is committed to the table.

    Recorded when a merge operation is committed to the table.

    updatePredicate, deletePredicate, and insertPredicate are DEPRECATED. Only use predicate, matchedPredicates, notMatchedPredicates and notMatchedBySourcePredicates to record the merge.

  11. val OP_RENAME_COLUMN: String

    Recorded when column is renamed

  12. val OP_RESTORE: String

    Recorded when restoring a Delta table to an older version.

  13. val OP_SET_TBLPROPERTIES: String

    Recorded when the table properties are set.

  14. val REORG_OPERATION_NAME: String

    operation name for REORG command

  15. val ROW_TRACKING_BACKFILL_OPERATION_NAME: String

    operation name for ROW TRACKING BACKFILL command

  16. val ZORDER_PARAMETER_KEY: String

    parameter key to indicate which columns to z-order by

  17. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  18. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  19. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  20. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  21. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  22. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  23. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  24. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  25. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  26. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  27. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  28. def predicatesToString(predicates: Seq[Expression]): Seq[String]

    Helper method to convert a sequence of command predicates in the form of an Expressions to a sequence of Strings so be stored in the commit info.

  29. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  30. def toString(): String
    Definition Classes
    AnyRef → Any
  31. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  32. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  33. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  34. object EmptyCommit extends Operation

    A commit without any actions.

    A commit without any actions. Could be used to force creation of new checkpoints.

  35. object ManualUpdate extends Operation
  36. object Merge extends Serializable
  37. object MergePredicate extends Serializable
  38. object VacuumEnd extends Serializable
  39. object VacuumStart extends Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped