Packages

  • package root
    Definition Classes
    root
  • package org
    Definition Classes
    root
  • package apache
    Definition Classes
    org
  • package spark
    Definition Classes
    apache
  • package sql
    Definition Classes
    spark
  • package catalyst

    Catalyst is a library for manipulating relational query plans.

    Catalyst is a library for manipulating relational query plans. All classes in catalyst are considered an internal API to Spark SQL and are subject to change between minor releases.

    Definition Classes
    sql
  • package optimizer
    Definition Classes
    catalyst
  • object JoinReorderDP extends PredicateHelper with Logging

    Reorder the joins using a dynamic programming algorithm.

    Reorder the joins using a dynamic programming algorithm. This implementation is based on the paper: Access Path Selection in a Relational Database Management System. https://dl.acm.org/doi/10.1145/582095.582099

    First we put all items (basic joined nodes) into level 0, then we build all two-way joins at level 1 from plans at level 0 (single items), then build all 3-way joins from plans at previous levels (two-way joins and single items), then 4-way joins ... etc, until we build all n-way joins and pick the best plan among them.

    When building m-way joins, we only keep the best plan (with the lowest cost) for the same set of m items. E.g., for 3-way joins, we keep only the best plan for items {A, B, C} among plans (A J B) J C, (A J C) J B and (B J C) J A. We also prune cartesian product candidates when building a new plan if there exists no join condition involving references from both left and right. This pruning strategy significantly reduces the search space. E.g., given A J B J C J D with join conditions A.k1 = B.k1 and B.k2 = C.k2 and C.k3 = D.k3, plans maintained for each level are as follows: level 0: p({A}), p({B}), p({C}), p({D}) level 1: p({A, B}), p({B, C}), p({C, D}) level 2: p({A, B, C}), p({B, C, D}) level 3: p({A, B, C, D}) where p({A, B, C, D}) is the final output plan.

    For cost evaluation, since physical costs for operators are not available currently, we use cardinalities and sizes to compute costs.

    Definition Classes
    optimizer
  • JoinPlan

case class JoinPlan(itemIds: Set[Int], plan: LogicalPlan, joinConds: ExpressionSet, planCost: Cost) extends Product with Serializable

Partial join order in a specific level.

itemIds

Set of item ids participating in this partial plan.

plan

The plan tree with the lowest cost for these items found so far.

joinConds

Join conditions included in the plan.

planCost

The cost of this plan tree is the sum of costs of all intermediate joins.

Linear Supertypes
Serializable, Product, Equals, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. JoinPlan
  2. Serializable
  3. Product
  4. Equals
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new JoinPlan(itemIds: Set[Int], plan: LogicalPlan, joinConds: ExpressionSet, planCost: Cost)

    itemIds

    Set of item ids participating in this partial plan.

    plan

    The plan tree with the lowest cost for these items found so far.

    joinConds

    Join conditions included in the plan.

    planCost

    The cost of this plan tree is the sum of costs of all intermediate joins.

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def betterThan(other: JoinPlan, conf: SQLConf): Boolean

    To identify the plan with smaller computational cost, we use the weighted geometric mean of ratio of rows and the ratio of sizes in bytes.

    To identify the plan with smaller computational cost, we use the weighted geometric mean of ratio of rows and the ratio of sizes in bytes.

    There are other ways to combine these values as a cost comparison function. Some of these, that we have experimented with, but have gotten worse result, than with the current one: 1) Weighted arithmetic mean of these two ratios - adding up fractions puts less emphasis on ratios between 0 and 1. Ratios 10 and 0.1 should be considered to be just as strong evidences in opposite directions. The arithmetic mean of these would be heavily biased towards the 10. 2) Absolute cost (cost = weight * rowCount + (1 - weight) * size) - when adding up two numeric measurements that have different units we can easily end up with one overwhelming the other.

  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  7. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  9. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  10. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  11. val itemIds: Set[Int]
  12. val joinConds: ExpressionSet
  13. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  14. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  15. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  16. val plan: LogicalPlan
  17. val planCost: Cost
  18. def productElementNames: Iterator[String]
    Definition Classes
    Product
  19. def rootCost(conf: SQLConf): Cost

    Get the cost of the root node of this plan tree.

  20. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  21. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  22. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  23. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from AnyRef

Inherited from Any

Ungrouped