class OptimisticTransaction extends OptimisticTransactionImpl with DeltaLogging
Used to perform a set of reads in a transaction and then commit a set of updates to the state of the log. All reads from the DeltaLog, MUST go through this instance rather than directly to the DeltaLog otherwise they will not be check for logical conflicts with concurrent updates.
This class is not thread-safe.
- Alphabetic
- By Inheritance
- OptimisticTransaction
- OptimisticTransactionImpl
- SQLMetricsReporting
- TransactionalWrite
- DeltaLogging
- DatabricksLogging
- DeltaProgressReporter
- Logging
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
-
new
OptimisticTransaction(deltaLog: DeltaLog)(implicit clock: Clock)
Creates a new OptimisticTransaction.
Creates a new OptimisticTransaction.
- deltaLog
The Delta Log for the table this transaction is modifying.
-
new
OptimisticTransaction(deltaLog: DeltaLog, snapshot: Snapshot)(implicit clock: Clock)
- deltaLog
The Delta Log for the table this transaction is modifying.
- snapshot
The snapshot that this transaction is reading at.
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
val
_spark: SparkSession
- Attributes
- protected
- Definition Classes
- OptimisticTransactionImpl
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
checkAndRetry(checkVersion: Long, actions: Seq[Action], attemptNumber: Int, commitIsolationLevel: IsolationLevel): Long
Looks at actions that have happened since the txn started and checks for logical conflicts with the read/writes.
Looks at actions that have happened since the txn started and checks for logical conflicts with the read/writes. If no conflicts are found, try to commit again otherwise, throw an exception.
- Attributes
- protected
- Definition Classes
- OptimisticTransactionImpl
-
implicit
val
clock: Clock
- Definition Classes
- OptimisticTransaction → OptimisticTransactionImpl
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
def
commit(actions: Seq[Action], op: Operation): Long
Modifies the state of the log by adding a new commit that is based on a read at the given
lastVersion.Modifies the state of the log by adding a new commit that is based on a read at the given
lastVersion. In the case of a conflict with a concurrent writer this method will throw an exception.- actions
Set of actions to commit
- op
Details of operation that is performing this transactional commit
- Definition Classes
- OptimisticTransactionImpl
- Annotations
- @throws( ... )
-
val
commitInfo: CommitInfo
- Attributes
- protected
- Definition Classes
- OptimisticTransactionImpl
-
val
commitStartNano: Long
- Attributes
- protected
- Definition Classes
- OptimisticTransactionImpl
-
val
committed: Boolean
Tracks if this transaction has already committed.
Tracks if this transaction has already committed.
- Attributes
- protected
- Definition Classes
- OptimisticTransactionImpl
-
val
deltaLog: DeltaLog
- Definition Classes
- OptimisticTransaction → OptimisticTransactionImpl → TransactionalWrite
-
def
doCommit(attemptVersion: Long, actions: Seq[Action], attemptNumber: Int, isolationLevel: IsolationLevel): Long
Commit
actionsusingattemptVersionversion number.Commit
actionsusingattemptVersionversion number. If detecting any conflicts, try to resolve logical conflicts and commit using a new version.- returns
the real version that was committed.
- Attributes
- protected
- Definition Classes
- OptimisticTransactionImpl
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
filterFiles(filters: Seq[Expression]): Seq[AddFile]
Returns files matching the given predicates.
Returns files matching the given predicates.
- Definition Classes
- OptimisticTransactionImpl
-
def
filterFiles(): Seq[AddFile]
Returns files matching the given predicates.
Returns files matching the given predicates.
- Definition Classes
- OptimisticTransactionImpl
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
getCommitter(outputPath: Path): DelayedCommitProtocol
- Attributes
- protected
- Definition Classes
- TransactionalWrite
-
def
getMetricsForOperation(operation: Operation): Map[String, String]
Get the metrics for an operation based on collected SQL Metrics and filtering out the ones based on the metric parameters for that operation.
Get the metrics for an operation based on collected SQL Metrics and filtering out the ones based on the metric parameters for that operation.
- Definition Classes
- SQLMetricsReporting
-
def
getNextAttemptVersion(previousAttemptVersion: Long): Long
Returns the next attempt version given the last attempted version
Returns the next attempt version given the last attempted version
- Attributes
- protected
- Definition Classes
- OptimisticTransactionImpl
-
def
getOperationMetrics(op: Operation): Option[Map[String, String]]
Return the operation metrics for the operation if it is enabled
Return the operation metrics for the operation if it is enabled
- Definition Classes
- OptimisticTransactionImpl
-
def
getPartitioningColumns(partitionSchema: StructType, output: Seq[Attribute], colsDropped: Boolean): Seq[Attribute]
- Attributes
- protected
- Definition Classes
- TransactionalWrite
-
def
getPrettyPartitionMessage(partitionValues: Map[String, String]): String
A helper function for pretty printing a specific partition directory.
A helper function for pretty printing a specific partition directory.
- Attributes
- protected
- Definition Classes
- OptimisticTransactionImpl
-
val
hasWritten: Boolean
- Attributes
- protected
- Definition Classes
- TransactionalWrite
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
- Attributes
- protected
- Definition Classes
- Logging
-
def
initializeLogIfNecessary(isInterpreter: Boolean): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
def
isTraceEnabled(): Boolean
- Attributes
- protected
- Definition Classes
- Logging
-
def
log: Logger
- Attributes
- protected
- Definition Classes
- Logging
-
def
logConsole(line: String): Unit
- Definition Classes
- DatabricksLogging
-
def
logDebug(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logDebug(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logError(msg: ⇒ String, throwable: Throwable): Unit
- Definition Classes
- OptimisticTransactionImpl → Logging
-
def
logError(msg: ⇒ String): Unit
- Definition Classes
- OptimisticTransactionImpl → Logging
-
def
logInfo(msg: ⇒ String): Unit
- Definition Classes
- OptimisticTransactionImpl → Logging
-
def
logInfo(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logName: String
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logWarning(msg: ⇒ String, throwable: Throwable): Unit
- Definition Classes
- OptimisticTransactionImpl → Logging
-
def
logWarning(msg: ⇒ String): Unit
- Definition Classes
- OptimisticTransactionImpl → Logging
-
def
makeOutputNullable(output: Seq[Attribute]): Seq[Attribute]
Makes the output attributes nullable, so that we don't write unreadable parquet files.
Makes the output attributes nullable, so that we don't write unreadable parquet files.
- Attributes
- protected
- Definition Classes
- TransactionalWrite
-
def
metadata: Metadata
Returns the metadata at the current point in the log.
Returns the metadata at the current point in the log.
- Definition Classes
- OptimisticTransactionImpl → TransactionalWrite
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
val
newMetadata: Option[Metadata]
Stores the updated metadata (if any) that will result from this txn.
Stores the updated metadata (if any) that will result from this txn.
- Attributes
- protected
- Definition Classes
- OptimisticTransactionImpl
-
def
normalizeData(data: Dataset[_], partitionCols: Seq[String]): (QueryExecution, Seq[Attribute])
Normalize the schema of the query, and return the QueryExecution to execute.
Normalize the schema of the query, and return the QueryExecution to execute. The output attributes of the QueryExecution may not match the attributes we return as the output schema. This is because streaming queries create
IncrementalExecution, which cannot be further modified. We can however have the Parquet writer use the physical plan fromIncrementalExecutionand the output schema provided through the attributes.- Attributes
- protected
- Definition Classes
- TransactionalWrite
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
def
postCommit(commitVersion: Long, commitActions: Seq[Action]): Unit
Perform post-commit operations
Perform post-commit operations
- Attributes
- protected
- Definition Classes
- OptimisticTransactionImpl
-
val
postCommitHooks: ArrayBuffer[PostCommitHook]
- Attributes
- protected
- Definition Classes
- OptimisticTransactionImpl
-
def
prepareCommit(actions: Seq[Action], op: Operation): Seq[Action]
Prepare for a commit by doing all necessary pre-commit checks and modifications to the actions.
Prepare for a commit by doing all necessary pre-commit checks and modifications to the actions.
- returns
The finalized set of actions.
- Attributes
- protected
- Definition Classes
- OptimisticTransactionImpl
-
val
protocol: Protocol
The protocol of the snapshot that this transaction is reading at.
The protocol of the snapshot that this transaction is reading at.
- Definition Classes
- OptimisticTransactionImpl → TransactionalWrite
-
val
readFiles: HashSet[AddFile]
Tracks specific files that have been seen by this transaction.
Tracks specific files that have been seen by this transaction.
- Attributes
- protected
- Definition Classes
- OptimisticTransactionImpl
-
val
readPredicates: ArrayBuffer[Expression]
Tracks the data that could have been seen by recording the partition predicates by which files have been queried by by this transaction.
Tracks the data that could have been seen by recording the partition predicates by which files have been queried by by this transaction.
- Attributes
- protected
- Definition Classes
- OptimisticTransactionImpl
-
val
readTxn: ArrayBuffer[String]
Tracks the appIds that have been seen by this transaction.
Tracks the appIds that have been seen by this transaction.
- Attributes
- protected
- Definition Classes
- OptimisticTransactionImpl
-
def
readVersion: Long
The version that this transaction is reading from.
The version that this transaction is reading from.
- Definition Classes
- OptimisticTransactionImpl
-
def
readWholeTable(): Unit
Mark the entire table as tainted by this transaction.
Mark the entire table as tainted by this transaction.
- Definition Classes
- OptimisticTransactionImpl
-
def
recordDeltaEvent(deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty, data: AnyRef = null): Unit
Used to record the occurrence of a single event or report detailed, operation specific statistics.
Used to record the occurrence of a single event or report detailed, operation specific statistics.
- Attributes
- protected
- Definition Classes
- DeltaLogging
-
def
recordDeltaOperation[A](deltaLog: DeltaLog, opType: String, tags: Map[TagDefinition, String] = Map.empty)(thunk: ⇒ A): A
Used to report the duration as well as the success or failure of an operation.
Used to report the duration as well as the success or failure of an operation.
- Attributes
- protected
- Definition Classes
- DeltaLogging
-
def
recordEvent(metric: MetricDefinition, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, trimBlob: Boolean = true): Unit
- Definition Classes
- DatabricksLogging
-
def
recordOperation[S](opType: OpType, opTarget: String = null, extraTags: Map[TagDefinition, String], isSynchronous: Boolean = true, alwaysRecordStats: Boolean = false, allowAuthTags: Boolean = false, killJvmIfStuck: Boolean = false, outputMetric: MetricDefinition = null, silent: Boolean = true)(thunk: ⇒ S): S
- Definition Classes
- DatabricksLogging
-
def
recordUsage(metric: MetricDefinition, quantity: Double, additionalTags: Map[TagDefinition, String] = Map.empty, blob: String = null, forceSample: Boolean = false, trimBlob: Boolean = true, silent: Boolean = false): Unit
- Definition Classes
- DatabricksLogging
-
def
registerPostCommitHook(hook: PostCommitHook): Unit
Register a hook that will be executed once a commit is successful.
Register a hook that will be executed once a commit is successful.
- Definition Classes
- OptimisticTransactionImpl
-
def
registerSQLMetrics(spark: SparkSession, metrics: Map[String, SQLMetric]): Unit
Register SQL metrics for an operation by appending the supplied metrics map to the operationSQLMetrics map.
Register SQL metrics for an operation by appending the supplied metrics map to the operationSQLMetrics map.
- Definition Classes
- SQLMetricsReporting
-
def
runPostCommitHooks(version: Long, committedActions: Seq[Action]): Unit
Executes the registered post commit hooks.
Executes the registered post commit hooks.
- Attributes
- protected
- Definition Classes
- OptimisticTransactionImpl
-
val
snapshot: Snapshot
- Definition Classes
- OptimisticTransaction → OptimisticTransactionImpl → TransactionalWrite
-
val
snapshotMetadata: Metadata
For new tables, fetch global configs as metadata.
For new tables, fetch global configs as metadata.
- Definition Classes
- OptimisticTransactionImpl
-
def
spark: SparkSession
- Attributes
- protected
- Definition Classes
- OptimisticTransactionImpl
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
val
txnStartNano: Long
- Attributes
- protected
- Definition Classes
- OptimisticTransactionImpl
-
def
txnVersion(id: String): Long
Returns the latest version that has committed for the idempotent transaction with given
id.Returns the latest version that has committed for the idempotent transaction with given
id.- Definition Classes
- OptimisticTransactionImpl
-
def
updateMetadata(metadata: Metadata): Unit
Records an update to the metadata that should be committed with this transaction.
Records an update to the metadata that should be committed with this transaction. Note that this must be done before writing out any files so that file writing and checks happen with the final metadata for the table.
IMPORTANT: It is the responsibility of the caller to ensure that files currently present in the table are still valid under the new metadata.
- Definition Classes
- OptimisticTransactionImpl
-
def
verifyNewMetadata(metadata: Metadata): Unit
- Attributes
- protected
- Definition Classes
- OptimisticTransactionImpl
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
def
withStatusCode[T](statusCode: String, defaultMessage: String, data: Map[String, Any] = Map.empty)(body: ⇒ T): T
Report a log to indicate some command is running.
Report a log to indicate some command is running.
- Definition Classes
- DeltaProgressReporter
-
def
writeFiles(data: Dataset[_], writeOptions: Option[DeltaOptions], isOptimize: Boolean): Seq[AddFile]
Writes out the dataframe after performing schema validation.
Writes out the dataframe after performing schema validation. Returns a list of actions to append these files to the reservoir.
- Definition Classes
- TransactionalWrite
-
def
writeFiles(data: Dataset[_], isOptimize: Boolean): Seq[AddFile]
- Definition Classes
- TransactionalWrite
-
def
writeFiles(data: Dataset[_], writeOptions: Option[DeltaOptions]): Seq[AddFile]
- Definition Classes
- TransactionalWrite
-
def
writeFiles(data: Dataset[_]): Seq[AddFile]
- Definition Classes
- TransactionalWrite