The column mapping mode of the target delta table.

The schema of the output attributes of the write queries that needs to collect statistics. The partition columns' definitions are not included in this schema.

The output attributes (outputAttributeSchema) that are replaced with table schema with the physical mapping information. NOTE: The partition columns' definitions are not included in this schema.

The snapshot that the scan is being generated on.

The statistic indexed column specification of the target delta table.

The schema of the target table of this statistics collection.

Traverses the statisticsSchema for the provided statisticsColumn and applies function to leaves.

Note, for values that are outside the domain of the partial function we keep the original column. If the caller wants to drop the column needs to explicitly return None.

Create a CachedDS instance for the given Dataset and the name.

Constructs a DataSkippingPredicate for isNotNull predicates.

Given the partition filters on the data, rewrite these filters by pointing to the metadata columns.

Helper method to check invariants in Delta code. Fails when running in tests, records a delta assertion event and logs a warning otherwise.

Gathers files that should be included in a scan based on the given predicates and limit. This will be called only when all predicates are on partitioning columns. Statistics about the amount of data that will be read are gathered and returned.

Gathers files that should be included in a scan based on the given predicates. Statistics about the amount of data that will be read are gathered and returned. Note, the statistics column that is added when keepNumRecords = true should NOT take into account DVs. Consumers of this method might commit the file. The semantics of the statistics need to be consistent across all files.

Returns a DataFrame for the given partition filters. The schema of returned DataFrame is nearly the same as AddFile, except that the stats field is parsed to a struct from a json string.

Get all the files in this table given the partition filter and the corresponding size of the scan.

Get all the files in this table.

Returns a Column that references the stats field data skipping should use

Given the partition and data filters, leverage data skipping statistics to find the set of files that need to be queried. Returns a tuple of the files and optionally the size of the scan that's generated if there were no filters, if there were only partition filters, and combined effect of partition and data filters respectively.

Get the files and number of records within each file, to perform limit pushdown.

Get AddFile (with stats) actions corresponding to given set of paths in the Snapshot. If a path doesn't exist in snapshot, it will be ignored and no AddFile will be returned for it.

Overload for convenience working with StatsColumn helpers

Returns an expression to access the given statistics for a specific column, or None if that stats column does not exist.

Overload for convenience working with StatsColumn helpers

Returns an expression to access the given statistics for a specific column, or a NULL literal expression if that column does not exist.

Used to record the occurrence of a single event or report detailed, operation specific statistics.

Used to report the duration as well as the success or failure of an operation on a deltaLog.

Used to report the duration as well as the success or failure of an operation on a tahoePath.

statCollectionLogicalSchema is the logical schema that is composed of all the columns that have the stats collected with our current table configuration.

statCollectionPhysicalSchema is the schema that is composed of all the columns that have the stats collected with our current table configuration.

Returns a struct column that can be used to collect statistics for the current schema of the table. The types we keep stats on must be consistent with DataSkippingReader.SkippingEligibleLiteral. If a column is missing from dataSchema (which will be filled with nulls), we will only collect the NULL_COUNT stats for it as the number of rows.

Returns schema of the statistics collected.

Drop any cached data for this Snapshot.

Sets the TIGHT_BOUNDS column to false and converts the logical nullCount to a tri-state nullCount. The nullCount states are the following: 1) For "all-nulls" columns we set the physical nullCount which is equal to the physical numRecords. 2) "no-nulls" columns remain unchanged, i.e. zero nullCount is the same for both physical and logical representations. 3) For "some-nulls" columns, we leave the existing value. In files with wide bounds, the nullCount in SOME_NULLs columns is considered unknown.

The file's state can transition back to tight when statistics are recomputed. In that case, TIGHT_BOUNDS is set back to true and nullCount back to the logical value.

Note, this function gets as input parsed statistics and returns a json document similarly to allFiles. To further match the behavior of allFiles we always return a column named stats instead of statsColName.

Returns an expression that can be used to check that the required statistics are present for a given file. If any required statistics are missing we must include the corresponding file.

NOTE: We intentionally choose to disable skipping for any file if any required stat is missing, because doing it that way allows us to check each stat only once (rather than once per use). Checking per-use would anyway only help for tables where the number of indexed columns has changed over time, producing add.stats_parsed records with differing schemas. That should be a rare enough case to not worry about optimizing for, given that the fix requires more complex skipping predicates that would penalize the common case.

All files with the statistics column dropped completely.

Returns a parsed and cached representation of files with statistics.

Report a log to indicate some command is running.