Packages

t

org.apache.spark.sql.delta

OptimisticTransactionImpl

trait OptimisticTransactionImpl extends TransactionalWrite with SQLMetricsReporting with DeltaLogging

Used to perform a set of reads in a transaction and then commit a set of updates to the state of the log. All reads from the DeltaLog, MUST go through this instance rather than directly to the DeltaLog otherwise they will not be check for logical conflicts with concurrent updates.

This trait is not thread-safe.

Linear Supertypes
SQLMetricsReporting, TransactionalWrite, DeltaLogging, DatabricksLogging, DeltaProgressReporter, Logging, AnyRef, Any
Known Subclasses
OptimisticTransaction
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. OptimisticTransactionImpl
  2. SQLMetricsReporting
  3. TransactionalWrite
  4. DeltaLogging
  5. DatabricksLogging
  6. DeltaProgressReporter
  7. Logging
  8. AnyRef
  9. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Abstract Value Members

  1. implicit abstract val clock: Clock
  2. abstract val deltaLog: DeltaLog
    Definition Classes
    OptimisticTransactionImpl → TransactionalWrite
  3. abstract val snapshot: Snapshot
    Definition Classes
    OptimisticTransactionImpl → TransactionalWrite

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. val _spark: SparkSession
    Attributes
    protected
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def checkAndRetry(checkVersion: Long, actions: Seq[Action], attemptNumber: Int, commitIsolationLevel: IsolationLevel): Long

    Looks at actions that have happened since the txn started and checks for logical conflicts with the read/writes.

    Looks at actions that have happened since the txn started and checks for logical conflicts with the read/writes. If no conflicts are found, try to commit again otherwise, throw an exception.

    Attributes
    protected
  7. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  8. def commit(actions: Seq[Action], op: Operation): Long

    Modifies the state of the log by adding a new commit that is based on a read at the given lastVersion.

    Modifies the state of the log by adding a new commit that is based on a read at the given lastVersion. In the case of a conflict with a concurrent writer this method will throw an exception.

    actions

    Set of actions to commit

    op

    Details of operation that is performing this transactional commit

    Annotations
    @throws( ... )
  9. val commitInfo: CommitInfo
    Attributes
    protected
  10. val commitStartNano: Long
    Attributes
    protected
  11. val committed: Boolean

    Tracks if this transaction has already committed.

    Tracks if this transaction has already committed.

    Attributes
    protected
  12. def doCommit(attemptVersion: Long, actions: Seq[Action], attemptNumber: Int, isolationLevel: IsolationLevel): Long

    Commit actions using attemptVersion version number.

    Commit actions using attemptVersion version number. If detecting any conflicts, try to resolve logical conflicts and commit using a new version.

    returns

    the real version that was committed.

    Attributes
    protected
  13. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  14. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  15. def filterFiles(filters: Seq[Expression]): Seq[AddFile]

    Returns files matching the given predicates.

  16. def filterFiles(): Seq[AddFile]

    Returns files matching the given predicates.

  17. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  18. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  19. def getCommitter(outputPath: Path): DelayedCommitProtocol
    Attributes
    protected
    Definition Classes
    TransactionalWrite
  20. def getMetricsForOperation(operation: Operation): Map[String, String]

    Get the metrics for an operation based on collected SQL Metrics and filtering out the ones based on the metric parameters for that operation.

    Get the metrics for an operation based on collected SQL Metrics and filtering out the ones based on the metric parameters for that operation.

    Definition Classes
    SQLMetricsReporting
  21. def getNextAttemptVersion(previousAttemptVersion: Long): Long

    Returns the next attempt version given the last attempted version

    Returns the next attempt version given the last attempted version

    Attributes
    protected
  22. def getOperationMetrics(op: Operation): Option[Map[String, String]]

    Return the operation metrics for the operation if it is enabled

  23. def getPartitioningColumns(partitionSchema: StructType, output: Seq[Attribute], colsDropped: Boolean): Seq[Attribute]
    Attributes
    protected
    Definition Classes
    TransactionalWrite
  24. def getPrettyPartitionMessage(partitionValues: Map[String, String]): String

    A helper function for pretty printing a specific partition directory.

    A helper function for pretty printing a specific partition directory.

    Attributes
    protected
  25. val hasWritten: Boolean
    Attributes
    protected
    Definition Classes
    TransactionalWrite
  26. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  27. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  28. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  29. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  30. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  31. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  32. def logConsole(line: String): Unit
    Definition Classes
    DatabricksLogging
  33. def logDebug(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  34. def logDebug(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  35. def logError(msg: ⇒ String, throwable: Throwable): Unit
    Definition Classes
    OptimisticTransactionImpl → Logging
  36. def logError(msg: ⇒ String): Unit
    Definition Classes
    OptimisticTransactionImpl → Logging
  37. def logInfo(msg: ⇒ String): Unit
    Definition Classes
    OptimisticTransactionImpl → Logging
  38. def logInfo(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  39. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  40. def logTrace(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  41. def logTrace(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  42. def logWarning(msg: ⇒ String, throwable: Throwable): Unit
    Definition Classes
    OptimisticTransactionImpl → Logging
  43. def logWarning(msg: ⇒ String): Unit
    Definition Classes
    OptimisticTransactionImpl → Logging
  44. def makeOutputNullable(output: Seq[Attribute]): Seq[Attribute]

    Makes the output attributes nullable, so that we don't write unreadable parquet files.

    Makes the output attributes nullable, so that we don't write unreadable parquet files.

    Attributes
    protected
    Definition Classes
    TransactionalWrite
  45. def metadata: Metadata

    Returns the metadata at the current point in the log.

    Returns the metadata at the current point in the log.

    Definition Classes
    OptimisticTransactionImpl → TransactionalWrite
  46. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  47. val newMetadata: Option[Metadata]

    Stores the updated metadata (if any) that will result from this txn.

    Stores the updated metadata (if any) that will result from this txn.

    Attributes
    protected
  48. def normalizeData(data: Dataset[_], partitionCols: Seq[String]): (QueryExecution, Seq[Attribute])

    Normalize the schema of the query, and return the QueryExecution to execute.

    Normalize the schema of the query, and return the QueryExecution to execute. The output attributes of the QueryExecution may not match the attributes we return as the output schema. This is because streaming queries create IncrementalExecution, which cannot be further modified. We can however have the Parquet writer use the physical plan from IncrementalExecution and the output schema provided through the attributes.

    Attributes
    protected
    Definition Classes
    TransactionalWrite
  49. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  50. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  51. def postCommit(commitVersion: Long, commitActions: Seq[Action]): Unit

    Perform post-commit operations

    Perform post-commit operations

    Attributes
    protected
  52. val postCommitHooks: ArrayBuffer[PostCommitHook]
    Attributes
    protected
  53. def prepareCommit(actions: Seq[Action], op: Operation): Seq[Action]

    Prepare for a commit by doing all necessary pre-commit checks and modifications to the actions.

    Prepare for a commit by doing all necessary pre-commit checks and modifications to the actions.

    returns

    The finalized set of actions.

    Attributes
    protected
  54. val protocol: Protocol

    The protocol of the snapshot that this transaction is reading at.

    The protocol of the snapshot that this transaction is reading at.

    Definition Classes
    OptimisticTransactionImpl → TransactionalWrite
  55. val readFiles: HashSet[AddFile]

    Tracks specific files that have been seen by this transaction.

    Tracks specific files that have been seen by this transaction.

    Attributes
    protected
  56. val readPredicates: ArrayBuffer[Expression]

    Tracks the data that could have been seen by recording the partition predicates by which files have been queried by by this transaction.

    Tracks the data that could have been seen by recording the partition predicates by which files have been queried by by this transaction.

    Attributes
    protected
  57. val readTxn: ArrayBuffer[String]

    Tracks the appIds that have been seen by this transaction.

    Tracks the appIds that have been seen by this transaction.

    Attributes
    protected
  58. def readVersion: Long

    The version that this transaction is reading from.

  59. def readWholeTable(): Unit

    Mark the entire table as tainted by this transaction.

  60. def recordDeltaEvent(deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty, data: AnyRef = null): Unit

    Used to record the occurrence of a single event or report detailed, operation specific statistics.

    Used to record the occurrence of a single event or report detailed, operation specific statistics.

    Attributes
    protected
    Definition Classes
    DeltaLogging
  61. def recordDeltaOperation[A](deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: ⇒ A): A

    Used to report the duration as well as the success or failure of an operation.

    Used to report the duration as well as the success or failure of an operation.

    Attributes
    protected
    Definition Classes
    DeltaLogging
  62. def recordEvent(metric: MetricDefinition, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit
    Definition Classes
    DatabricksLogging
  63. def recordOperation[S](opType: OpType, opTarget: String = null, extraTags: Map[TagDefinition, String], isSynchronous: Boolean = true, alwaysRecordStats: Boolean = false, allowAuthTags: Boolean = false, killJvmIfStuck: Boolean = false, outputMetric: MetricDefinition = null, silent: Boolean = true)(thunk: ⇒ S): S
    Definition Classes
    DatabricksLogging
  64. def recordUsage(metric: MetricDefinition, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit
    Definition Classes
    DatabricksLogging
  65. def registerPostCommitHook(hook: PostCommitHook): Unit

    Register a hook that will be executed once a commit is successful.

  66. def registerSQLMetrics(spark: SparkSession, metrics: Map[String, SQLMetric]): Unit

    Register SQL metrics for an operation by appending the supplied metrics map to the operationSQLMetrics map.

    Register SQL metrics for an operation by appending the supplied metrics map to the operationSQLMetrics map.

    Definition Classes
    SQLMetricsReporting
  67. def runPostCommitHooks(version: Long, committedActions: Seq[Action]): Unit

    Executes the registered post commit hooks.

    Executes the registered post commit hooks.

    Attributes
    protected
  68. val snapshotMetadata: Metadata

    For new tables, fetch global configs as metadata.

  69. def spark: SparkSession
    Attributes
    protected
  70. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  71. def toString(): String
    Definition Classes
    AnyRef → Any
  72. val txnStartNano: Long
    Attributes
    protected
  73. def txnVersion(id: String): Long

    Returns the latest version that has committed for the idempotent transaction with given id.

  74. def updateMetadata(metadata: Metadata): Unit

    Records an update to the metadata that should be committed with this transaction.

    Records an update to the metadata that should be committed with this transaction. Note that this must be done before writing out any files so that file writing and checks happen with the final metadata for the table.

    IMPORTANT: It is the responsibility of the caller to ensure that files currently present in the table are still valid under the new metadata.

  75. def verifyNewMetadata(metadata: Metadata): Unit
    Attributes
    protected
  76. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  77. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  78. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  79. def withStatusCode[T](statusCode: String, defaultMessage: String, data: Map[String, Any] = Map.empty)(body: ⇒ T): T

    Report a log to indicate some command is running.

    Report a log to indicate some command is running.

    Definition Classes
    DeltaProgressReporter
  80. def writeFiles(data: Dataset[_], writeOptions: Option[DeltaOptions], isOptimize: Boolean): Seq[AddFile]

    Writes out the dataframe after performing schema validation.

    Writes out the dataframe after performing schema validation. Returns a list of actions to append these files to the reservoir.

    Definition Classes
    TransactionalWrite
  81. def writeFiles(data: Dataset[_], isOptimize: Boolean): Seq[AddFile]
    Definition Classes
    TransactionalWrite
  82. def writeFiles(data: Dataset[_], writeOptions: Option[DeltaOptions]): Seq[AddFile]
    Definition Classes
    TransactionalWrite
  83. def writeFiles(data: Dataset[_]): Seq[AddFile]
    Definition Classes
    TransactionalWrite

Inherited from SQLMetricsReporting

Inherited from TransactionalWrite

Inherited from DeltaLogging

Inherited from DatabricksLogging

Inherited from DeltaProgressReporter

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped