Tracks actions within the transaction, will commit along with the passed-in actions in the commit function.

Looks at actions that have happened since the txn started and checks for logical conflicts with the read/writes. Resolve conflicts and returns a tuple representing the commit version to attempt next and the commit summary which we need to commit.

Checks if the passed-in actions have internal SetTransaction conflicts, will throw exceptions in case of conflicts. This function will also remove duplicated SetTransactions.

If the operation assigns or modifies column default values, this method checks that the corresponding table feature is enabled and throws an error if not.

Whether to check unsupported data type when updating the table schema

Modifies the state of the log by adding a new commit that is based on a read at readVersion. In the case of a conflict with a concurrent writer this method will throw an exception.

Modifies the state of the log by adding a new commit that is based on a read at readVersion. In the case of a conflict with a concurrent writer this method will throw an exception.

Tracks the start time since we started trying to write a particular commit. Used for logging duration of retried transactions.

The transaction commit end time.

Modifies the state of the log by adding a new commit that is based on a read at readVersion. In the case of a conflict with a concurrent writer this method will throw an exception.

Also skips creating the commit if the configured IsolationLevel doesn't need us to record the commit from correctness perspective.

Create a large commit on the Delta log by directly writing an iterator of FileActions to the LogStore. This function only commits the next possible version and will not check whether the commit is retry-able. If the next version has already been committed, then this function will fail. This bypasses all optimistic concurrency checks. We assume that transaction conflicts should be rare because this method is typically used to create new tables (e.g. CONVERT TO DELTA) or apply some commands which rarely receive other transactions (e.g. CLONE/RESTORE). In addition, the expectation is that the list of actions performed by the transaction remains an iterator and is never materialized, given the nature of a large commit potentially touching many files. The nonProtocolMetadataActions parameter should only contain non-{protocol, metadata} actions only. If the protocol of table needs to be updated, it should be passed in the newProtocolOpt parameter.

The transaction commit start time.

Tracks if this transaction has already committed.

If there is any string partition column and there are constraints defined, add a projection to convert empty string to null for that column. The empty strings will be converted to null eventually even without this convert, but we want to do this earlier before check constraints so that empty strings are correctly rejected. Note that this should not cause the downstream logic in FileFormatWriter to add duplicate conversions because the logic there checks the partition column using the original plan's output. When the plan is modified with additional projections, the partition column check won't match and will not add more conversion.

Helper method to check invariants in Delta code. Fails when running in tests, records a delta assertion event and logs a warning otherwise.

Disable the check that ensures that all files with DVs added have tightBounds set to false.

This is necessary when recomputing the stats on a table with DVs.

Commit actions using attemptVersion version number. Throws a FileAlreadyExistsException if any conflicts are detected.

Commit the txn represented by currentTransactionInfo using attemptVersion version number. If there are any conflicts that are found, we will retry a fixed number of times.

Contains the execution instrumentation set via thread-local. No-op by default.

Returns aDeltaScan based on the given partition filters, projections and limits.

Returns aDeltaScan based on the given filters.

Returns a DataFrame for the given partition filters. The schema of returned DataFrame is nearly the same as AddFile, except that the stats field is parsed to a struct from a json string.

Returns files within the given partitions.

partitions is a set of the partitionValues stored in AddFiles. This means they refer to the physical column names, and values are stored as strings.

Generates a timestamp which is greater than the commit timestamp of the last snapshot. Note that this is only needed when the feature inCommitTimestamps is enabled.

Must make sure that deletion vectors are never added to a table where that isn't allowed. Note, statistics recomputation is still allowed even though DVs might be currently disabled.

This method returns a function that can be used to validate a single Action.

Returns the conflicting commit information

Returns the DeltaScanGenerator for the given log, which will be used to generate DeltaScans. Every time this method is called on a log, the returned generator generator will read a snapshot that is pinned on the first access for that log.

Internally, if the given log is the same as the log associated with this transaction, then it returns this transaction, otherwise it will return a snapshot of given log

Returns the metric with name registered for the given transaction if it exists.

Get the metrics for an operation based on collected SQL Metrics and filtering out the ones based on the metric parameters for that operation.

Return the pair of optional stats tracker and stats collection class

Return a tuple of (outputStatsCollectionSchema, statsCollectionSchema). outputStatsCollectionSchema is the data source schema from DataFrame used for stats collection. It contains the columns in the DataFrame output, excluding the partition columns. tableStatsCollectionSchema is the schema to collect stats for. It contains the columns in the table schema, excluding the partition columns. Note: We only collect NULL_COUNT stats (as the number of rows) for the columns in statsCollectionSchema but missing in outputStatsCollectionSchema

True if this transaction is a blind append. This is only valid after commit.

Makes the output attributes nullable, so that we don't write unreadable parquet files.

Replace the output attributes with the physical mapping information.

Returns the metadata for this transaction. The metadata refers to the metadata of the snapshot at the transaction's read version unless updated during the transaction.

Stores the updated metadata (if any) that will result from this txn.

This is just one way to change metadata. New metadata can also be added during commit from actions. But metadata should *not* be updated via both paths.

Stores the updated protocol (if any) that will result from this txn.

Normalize the schema of the query, and return the QueryExecution to execute. If the table has generated columns and users provide these columns in the output, we will also return constraints that should be respected. If any constraints are returned, the caller should apply these constraints when writing data.

Note: The output attributes of the QueryExecution may not match the attributes we return as the output schema. This is because streaming queries create IncrementalExecution, which cannot be further modified. We can however have the Parquet writer use the physical plan from IncrementalExecution and the output schema provided through the attributes.

The set of distinct partitions that contain added files by current transaction.

Returns a tuple of (data, partition schema). For CDC writes, a is_cdc column is added to the data and is_cdc=true/false is added to the front of the partition schema.

Checks if the new schema contains any CDC columns (which is invalid) and throws the appropriate error

Prepare for a commit by doing all necessary pre-commit checks and modifications to the actions.

The protocol of the snapshot that this transaction is reading at.

Tracks specific files that have been seen by this transaction.

Tracks the data that could have been seen by recording the partition predicates by which files have been queried by this transaction.

Tracks the first-access snapshots of other Delta logs read by this transaction. The snapshots are keyed by the log's unique id.

Whether the whole table was read during the transaction.

Tracks the appIds that have been seen by this transaction.

Used to record the occurrence of a single event or report detailed, operation specific statistics.

Used to report the duration as well as the success or failure of an operation on a deltaLog.

Used to report the duration as well as the success or failure of an operation on a tahoePath.

Register SQL metrics for an operation by appending the supplied metrics map to the operationSQLMetrics map.

This method registers the table with the commit-owner via the CommitOwnerClient if the table is transitioning from file-system based table to managed-commit table.

Executes the registered post commit hooks.

Sets needsCheckpoint if we should checkpoint the version that has just been committed.

Whether to skip recording the commit in DeltaLog

The snapshot that the scan is being generated on.

The transaction start time.

Update the table now that the commit has been made, and write a checkpoint.

Records an update to the metadata that should be committed with this transaction. Note that this must be done before writing out any files so that file writing and checks happen with the final metadata for the table.

IMPORTANT: It is the responsibility of the caller to ensure that files currently present in the table are still valid under the new metadata.

Records an update to the metadata that should be committed with this transaction and when this transaction is logically creating a new table, e.g. replacing a previous table with new metadata. Note that this must be done before writing out any files so that file writing and checks happen with the final metadata for the table. IMPORTANT: It is the responsibility of the caller to ensure that files currently present in the table are still valid under the new metadata.

Records an update to the metadata that should be committed with this transaction and when this transaction is attempt to overwrite the data and schema using .mode('overwrite') and .option('overwriteSchema', true). REPLACE the table is not considered in this category, because that is logically equivalent to DROP and RECREATE the table.

Do the actual checks and works to update the metadata and save it into the newMetadata field, which will be added to the actions to commit in prepareCommit.

This method makes the necessary changes to Metadata based on ICT: If ICT is getting enabled as part of this commit, then it updates the Metadata with the ICT enablement information.

This method makes the necessary changes to Metadata based on managed-commit: If the table is being converted from file-system to managed commits, then it registers the table with the commit-owner and updates the Metadata with the necessary configuration information from the commit-owner.

This updates the protocol for the table with a given protocol. Note that the protocol set by this method can be overwritten by other methods, such as updateMetadata.

Report a log to indicate some command is running.

Writes the json actions provided to the commit file corresponding to attemptVersion. If managed-commits are enabled, this method must return a non-empty Commit since we can't guess it from the FileSystem.

Writes out the dataframe after performing schema validation. Returns a list of actions to append these files to the reservoir.