io.delta.kernel.internal.skipping

Class StatsSchemaHelper

Object

io.delta.kernel.internal.skipping.StatsSchemaHelper

public class StatsSchemaHelper
extends Object

Provides information and utilities for statistics columns given a table schema. Specifically, it is used to:

1. Get the expected statistics schema given a table schema
2. Check if a Literal or Column is skipping eligible
3. Get the statistics column for a given stat type and logical column

Constructor Summary

Constructors

Constructor and Description
StatsSchemaHelper(StructType dataSchema)

Method Summary

Modifier and Type	Method and Description
Tuple2<Column,java.util.Optional<Expression>>	getMaxColumn(Column column) Given a logical column in the data schema provided when creating this, return the corresponding MAX column and an optional column adjustment expression from the statistic schema that stores the MAX values for the provided logical column.
Tuple2<Column,java.util.Optional<Expression>>	getMinColumn(Column column) Given a logical column in the data schema provided when creating this, return the corresponding MIN column and an optional column adjustment expression from the statistic schema that stores the MIN values for the provided logical column.
Column	getNullCountColumn(Column column) Given a logical column in the data schema provided when creating this, return the corresponding NULL_COUNT column in the statistic schema that stores the null count values for the provided logical column.
Column	getNumRecordsColumn() Returns the NUM_RECORDS column in the statistic schema
static StructType	getStatsSchema(StructType dataSchema) Returns the expected statistics schema given a table schema.
static boolean	isSkippingEligibleLiteral(Literal literal) Returns true if the given literal is skipping-eligible.
boolean	isSkippingEligibleMinMaxColumn(Column column) Returns true if the given column is skipping-eligible using min/max statistics.
boolean	isSkippingEligibleNullCountColumn(Column column) Returns true if the given column is skipping-eligible using null count statistics.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StatsSchemaHelper

public StatsSchemaHelper(StructType dataSchema)

Method Detail

isSkippingEligibleLiteral

public static boolean isSkippingEligibleLiteral(Literal literal)

Returns true if the given literal is skipping-eligible. Delta tracks min/max stats for a limited set of data types and only literals of those types are skipping eligible.

getStatsSchema

public static StructType getStatsSchema(StructType dataSchema)

Returns the expected statistics schema given a table schema.

Here is an example of a data schema along with the schema of the statistics that would be collected.

Data Schema: {{{ |-- a: struct (nullable = true) | |-- b: struct (nullable = true) | | |-- c: long (nullable = true) }}}

Collected Statistics: {{{ |-- stats: struct (nullable = true) | |-- numRecords: long (nullable = false) | |-- minValues: struct (nullable = false) | | |-- a: struct (nullable = false) | | | |-- b: struct (nullable = false) | | | | |-- c: long (nullable = true) | |-- maxValues: struct (nullable = false) | | |-- a: struct (nullable = false) | | | |-- b: struct (nullable = false) | | | | |-- c: long (nullable = true) | |-- nullCount: struct (nullable = false) | | |-- a: struct (nullable = false) | | | |-- b: struct (nullable = false) | | | | |-- c: long (nullable = true) }}}

getMinColumn

public Tuple2<Column,java.util.Optional<Expression>> getMinColumn(Column column)

Given a logical column in the data schema provided when creating this, return the corresponding MIN column and an optional column adjustment expression from the statistic schema that stores the MIN values for the provided logical column.

Parameters:

column - the logical column name.

Returns:

a tuple of the MIN column and an optional adjustment expression.

getMaxColumn

public Tuple2<Column,java.util.Optional<Expression>> getMaxColumn(Column column)

Given a logical column in the data schema provided when creating this, return the corresponding MAX column and an optional column adjustment expression from the statistic schema that stores the MAX values for the provided logical column.

Parameters:

column - the logical column name.

Returns:

a tuple of the MAX column and an optional adjustment expression.

getNullCountColumn

public Column getNullCountColumn(Column column)

Given a logical column in the data schema provided when creating this, return the corresponding NULL_COUNT column in the statistic schema that stores the null count values for the provided logical column.

getNumRecordsColumn

public Column getNumRecordsColumn()

Returns the NUM_RECORDS column in the statistic schema

isSkippingEligibleMinMaxColumn

public boolean isSkippingEligibleMinMaxColumn(Column column)

Returns true if the given column is skipping-eligible using min/max statistics. This means the column exists, is a leaf column, and is of a skipping-eligible data-type.

isSkippingEligibleNullCountColumn

public boolean isSkippingEligibleNullCountColumn(Column column)

Returns true if the given column is skipping-eligible using null count statistics. This means the column exists and is a leaf column as we only collect stats for leaf columns.