class DataFiltersBuilder extends AnyRef
Builds the data filters for data skipping.
- Alphabetic
- By Inheritance
- DataFiltersBuilder
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new DataFiltersBuilder(spark: SparkSession, dataSkippingType: DeltaDataSkippingType)
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- def apply(dataFilter: Expression): Option[DataSkippingPredicate]
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @native()
- val dataSkippingType: DeltaDataSkippingType
- Attributes
- protected
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable])
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- def rewriteDataFiltersAsPartitionLike(clusteringColumns: Seq[String], expr: Expression): Option[DataSkippingPredicate]
Rewrites the given expression as a partition-like expression if possible: 1.
Rewrites the given expression as a partition-like expression if possible: 1. Rewrite the attribute references in the expression to reference the collected min stats on the attribute reference's column. 2. Construct an expression that returns true if any of the referenced columns are not partition-like on a given file. The rewritten expression is a union of the above expressions: a file is read if it's either not partition-like on any of the columns or if the rewritten expression evaluates to true.
- clusteringColumns
The columns that are used for clustering.
- expr
The data filtering expression to rewrite.
- returns
If the expression is safe to rewrite, return the rewritten expression. Otherwise, return None.
- val spark: SparkSession
- Attributes
- protected
- val statsProvider: StatsProvider
- Attributes
- protected
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- object SkippingEligibleExpression
An extractor that matches expressions that are eligible for data skipping predicates.
An extractor that matches expressions that are eligible for data skipping predicates.
- returns
A tuple of 1) column name referenced in the expression, 2) date type for the expression, 3) DataSkippingPredicateBuilder that builds the data skipping predicate for the expression, if the given expression is eligible. Otherwise, return None.