Packages

t

org.apache.spark.sql.delta

PreprocessTableWithDVs

trait PreprocessTableWithDVs extends SubqueryTransformerHelper

Plan transformer to inject a filter that removes the rows marked as deleted according to deletion vectors. For tables with no deletion vectors, this transformation has no effect.

It modifies for plan for tables with deletion vectors as follows: Before rule: <Parent Node> -> Delta Scan (key, value).

  • Here we are reading key, value columns from the Delta table After rule: <Parent Node> -> Project(key, value) -> Filter (skip_row == 0) -> Delta Scan (key, value, skip_row)
  • Here we insert a new column skip_row in Delta scan. This value is populated by the Parquet reader using the DV corresponding to the Parquet file read (See DeltaParquetFileFormat) and it contains 0 if we want to keep the row.
  • Filter created filters out rows with skip_row equals to 0
  • And at the end we have a Project to keep the plan node output same as before the rule is applied.
Linear Supertypes
SubqueryTransformerHelper, AnyRef, Any
Known Subclasses
PreprocessTableWithDVsStrategy
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. PreprocessTableWithDVs
  2. SubqueryTransformerHelper
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  10. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  12. def isSubqueryRoot(plan: LogicalPlan): Boolean

    Is the give plan a subquery root.

    Is the give plan a subquery root.

    Definition Classes
    SubqueryTransformerHelper
  13. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  14. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  15. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  16. def preprocessTablesWithDVs(plan: LogicalPlan): LogicalPlan
  17. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  18. def toString(): String
    Definition Classes
    AnyRef → Any
  19. def transformWithSubqueries(plan: LogicalPlan)(rule: PartialFunction[LogicalPlan, LogicalPlan]): LogicalPlan

    Transform all nodes matched by the rule in the query plan rooted at given plan.

    Transform all nodes matched by the rule in the query plan rooted at given plan. It traverses the tree starting from the leaves, whenever a SubqueryExpression expression is encountered, given rule is applied to the subquery plan plan in SubqueryExpression starting from the plan root until leaves.

    This is slightly different behavior compared to QueryPlan.transformUpWithSubqueries or QueryPlan.transformDownWithSubqueries

    It requires that the given plan already gone through OptimizeSubqueries and the root node denoting a subquery is removed and optimized appropriately.

    Definition Classes
    SubqueryTransformerHelper
  20. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  21. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  22. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from SubqueryTransformerHelper

Inherited from AnyRef

Inherited from Any

Ungrouped