Packages

•  package root

  Definition Classes

  root

•  package org

  Definition Classes

  root

•  package apache

  Definition Classes

  org

•  package spark

  Definition Classes

  apache

•  package sql

  Definition Classes

  spark

•  package catalyst

  Catalyst is a library for manipulating relational query plans.

  Catalyst is a library for manipulating relational query plans. All classes in catalyst are considered an internal API to Spark SQL and are subject to change between minor releases.

  Definition Classes

  sql

•  package analysis

  Provides a logical query plan Analyzer and supporting classes for performing analysis.

  Provides a logical query plan Analyzer and supporting classes for performing analysis. Analysis consists of translating UnresolvedAttributes and UnresolvedRelations into fully typed objects using information in a schema Catalog.

  Definition Classes

  catalyst

•  object ResolveLateralColumnAliasReference extends Rule[LogicalPlan]

  This rule is the second phase to resolve lateral column alias.

  This rule is the second phase to resolve lateral column alias.

  Resolve lateral column alias, which references the alias defined previously in the SELECT list. Plan-wise, it handles two types of operators: Project and Aggregate. - in Project, pushing down the referenced lateral alias into a newly created Project, resolve the attributes referencing these aliases - in Aggregate, inserting the Project node above and falling back to the resolution of Project.

  The whole process is generally divided into two phases: 1) recognize resolved lateral alias, wrap the attributes referencing them with LateralColumnAliasReference 2) when the whole operator is resolved, or contains Window but have all other resolved, For Project, it unwrap LateralColumnAliasReference, further resolves the attributes and push down the referenced lateral aliases. For Aggregate, it goes through the whole aggregation list, extracts the aggregation expressions and grouping expressions to keep them in this Aggregate node, and add a Project above with the original output. It doesn't do anything on LateralColumnAliasReference, but completely leave it to the Project in the future turns of this rule.

  ** Example for Project: Before rewrite: Project [age AS a, 'a + 1] +- Child

  After phase 1: Project [age AS a, lca(a) + 1] +- Child

  After phase 2: Project [a, a + 1] +- Project [child output, age AS a] +- Child

  ** Example for Aggregate: Before rewrite: Aggregate [dept#14] [dept#14 AS a#12, 'a + 1, avg(salary#16) AS b#13, 'b + avg(bonus#17)] +- Child [dept#14,name#15,salary#16,bonus#17]

  After phase 1: Aggregate [dept#14] [dept#14 AS a#12, lca(a) + 1, avg(salary#16) AS b#13, lca(b) + avg(bonus#17)] +- Child [dept#14,name#15,salary#16,bonus#17]

  After phase 2: Project [dept#14 AS a#12, lca(a) + 1, avg(salary)#26 AS b#13, lca(b) + avg(bonus)#27] +- Aggregate [dept#14] [avg(salary#16) AS avg(salary)#26, avg(bonus#17) AS avg(bonus)#27,dept#14] +- Child [dept#14,name#15,salary#16,bonus#17]

  Now the problem falls back to the lateral alias resolution in Project. After future rounds of this rule: Project [a#12, a#12 + 1, b#13, b#13 + avg(bonus)#27] +- Project [dept#14 AS a#12, avg(salary)#26 AS b#13] +- Aggregate [dept#14] [avg(salary#16) AS avg(salary)#26, avg(bonus#17) AS avg(bonus)#27, dept#14] +- Child [dept#14,name#15,salary#16,bonus#17]

  ** Example for Window: Query: select dept as d, sum(salary) as s, avg(s) over (partition by s order by d) as avg from employee group by dept

  After phase 1: 'Aggregate [dept#17], [dept#17 AS d#15, sum(salary#19) AS s#16L, avg(lca(s#16L)) windowspecdefinition(lca(s#16L), lca(d#15) ASC NULLS FIRST, specifiedwindowframe(..)) AS avg#25] +- Relation spark_catalog.default.employee[dept#17,name#18,salary#19,bonus#20,properties#21] It is similar to a regular Aggregate. All expressions in it are resolved, but itself is unresolved due to the Window expression. The rule allows appliction on this case.

  After phase 2: 'Project [dept#17 AS d#15, sum(salary)#26L AS s#16L, avg(lca(s#16L)) windowspecdefinition(lca(s#16L), lca(d#15) ASC NULLS FIRST, specifiedwindowframe(..)) AS avg#25] +- Aggregate [dept#17], [dept#17, sum(salary#19) AS sum(salary)#26L] +- Relation spark_catalog.default.employee[dept#17,name#18,salary#19,bonus#20,properties#21] Same as Aggregate, it extracts grouping expressions and aggregate functions. Window expressions are completely lifted up to upper Project, free from the current Aggregate.

  Then this rule will apply on the Project, adding another Project below. Till this phase, all lateral column alias references have been resolved and removed. Finally, rule ExtractWindowExpressions will apply on the top Project with window expressions. It is guaranteed that ResolveLateralColumnAliasReference is applied before ExtractWindowExpressions.

  Definition Classes

  analysis

• AliasEntry

org.apache.spark.sql.catalyst.analysis.ResolveLateralColumnAliasReference

AliasEntry