Packages

•  package root

  Definition Classes

  root

•  package org

  Definition Classes

  root

•  package apache

  Definition Classes

  org

•  package spark

  Definition Classes

  apache

•  package sql

  Definition Classes

  spark

•  package catalyst

  Catalyst is a library for manipulating relational query plans.

  Catalyst is a library for manipulating relational query plans. All classes in catalyst are considered an internal API to Spark SQL and are subject to change between minor releases.

  Definition Classes

  sql

•  package optimizer

  Definition Classes

  catalyst

•  object MergeScalarSubqueries extends Rule[LogicalPlan]

  This rule tries to merge multiple non-correlated ScalarSubquerys to compute multiple scalar values once.

  This rule tries to merge multiple non-correlated ScalarSubquerys to compute multiple scalar values once.

  The process is the following: - While traversing through the plan each ScalarSubquery plan is tried to merge into the cache of already seen subquery plans. If merge is possible then cache is updated with the merged subquery plan, if not then the new subquery plan is added to the cache. During this first traversal each ScalarSubquery expression is replaced to a temporal ScalarSubqueryReference reference pointing to its cached version. The cache uses a flag to keep track of if a cache entry is a result of merging 2 or more plans, or it is a plan that was seen only once. Merged plans in the cache get a "Header", that contains the list of attributes form the scalar return value of a merged subquery. - A second traversal checks if there are merged subqueries in the cache and builds a WithCTE node from these queries. The CTERelationDef nodes contain the merged subquery in the following form: Project(Seq(CreateNamedStruct(name1, attribute1, ...) AS mergedValue), mergedSubqueryPlan) and the definitions are flagged that they host a subquery, that can return maximum one row. During the second traversal ScalarSubqueryReference expressions that pont to a merged subquery is either transformed to a GetStructField(ScalarSubquery(CTERelationRef(...))) expression or restored to the original ScalarSubquery.

  Eg. the following query:

  SELECT (SELECT avg(a) FROM t), (SELECT sum(b) FROM t)

  is optimized from:

  Optimized Logical Plan

  Project [scalar-subquery#242 [] AS scalarsubquery()#253, scalar-subquery#243 [] AS scalarsubquery()#254L] : :- Aggregate [avg(a#244) AS avg(a)#247] : : +- Project [a#244] : : +- Relation default.t[a#244,b#245] parquet : +- Aggregate [sum(a#251) AS sum(a)#250L] : +- Project [a#251] : +- Relation default.t[a#251,b#252] parquet +- OneRowRelation

  to:

  Optimized Logical Plan

  Project [scalar-subquery#242 [].avg(a) AS scalarsubquery()#253, scalar-subquery#243 [].sum(a) AS scalarsubquery()#254L] : :- Project [named_struct(avg(a), avg(a)#247, sum(a), sum(a)#250L) AS mergedValue#260] : : +- Aggregate [avg(a#244) AS avg(a)#247, sum(a#244) AS sum(a)#250L] : : +- Project [a#244] : : +- Relation default.t[a#244,b#245] parquet : +- Project [named_struct(avg(a), avg(a)#247, sum(a), sum(a)#250L) AS mergedValue#260] : +- Aggregate [avg(a#244) AS avg(a)#247, sum(a#244) AS sum(a)#250L] : +- Project [a#244] : +- Relation default.t[a#244,b#245] parquet +- OneRowRelation

  Physical Plan

  *(1) Project [Subquery scalar-subquery#242, [id=#125].avg(a) AS scalarsubquery()#253, ReusedSubquery Subquery scalar-subquery#242, [id=#125].sum(a) AS scalarsubquery()#254L] : :- Subquery scalar-subquery#242, [id=#125] : : +- *(2) Project [named_struct(avg(a), avg(a)#247, sum(a), sum(a)#250L) AS mergedValue#260] : : +- *(2) HashAggregate(keys=[], functions=[avg(a#244), sum(a#244)], output=[avg(a)#247, sum(a)#250L]) : : +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [id=#120] : : +- *(1) HashAggregate(keys=[], functions=[partial_avg(a#244), partial_sum(a#244)], output=[sum#262, count#263L, sum#264L]) : : +- *(1) ColumnarToRow : : +- FileScan parquet default.t[a#244] ... : +- ReusedSubquery Subquery scalar-subquery#242, [id=#125] +- *(1) Scan OneRowRelation[]

  Definition Classes

  optimizer

• Header

org.apache.spark.sql.catalyst.optimizer.MergeScalarSubqueries

Header