object GeneratedColumnAnalyzer extends Analyzer
Analyzer for processing generated column expressions using built-in functions only.
- Alphabetic
- By Inheritance
- GeneratedColumnAnalyzer
- Analyzer
- ColumnResolutionHelper
- SQLConfHelper
- CheckAnalysis
- QueryErrorsBase
- DataTypeErrorsBase
- LookupCatalog
- PredicateHelper
- AliasHelper
- RuleExecutor
- Logging
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Type Members
- case class Batch(name: String, strategy: Strategy, rules: Rule[TreeType]*) extends Product with Serializable
A batch of rules.
A batch of rules.
- Attributes
- protected[catalyst]
- Definition Classes
- RuleExecutor
- case class FixedPoint(maxIterations: Int, errorOnExceed: Boolean = false, maxIterationsSetting: String = null) extends Strategy with Product with Serializable
A strategy that runs until fix point or maxIterations times, whichever comes first.
A strategy that runs until fix point or maxIterations times, whichever comes first. Especially, a FixedPoint(1) batch is supposed to run only once.
- Definition Classes
- RuleExecutor
- abstract class Strategy extends AnyRef
An execution strategy for rules that indicates the maximum number of executions.
An execution strategy for rules that indicates the maximum number of executions. If the execution reaches fix point (i.e. converge) before maxIterations, it will stop.
- Definition Classes
- RuleExecutor
Value Members
- object AddMetadataColumns extends Rule[LogicalPlan]
Adds metadata columns to output for child relations when nodes are missing resolved attributes.
Adds metadata columns to output for child relations when nodes are missing resolved attributes.
References to metadata columns are resolved using columns from LogicalPlan.metadataOutput, but the relation's output does not include the metadata columns until the relation is replaced. Unless this rule adds metadata to the relation's output, the analyzer will detect that nothing produces the columns.
This rule only adds metadata columns when a node is resolved but is missing input from its children. This ensures that metadata columns are not added to the plan unless they are used. By checking only resolved nodes, this ensures that * expansion is already done so that metadata columns are not accidentally selected by *. This rule resolves operators downwards to avoid projecting away metadata columns prematurely.
- Definition Classes
- Analyzer
- object ExtractGenerator extends Rule[LogicalPlan]
Extracts Generator from the projectList of a Project operator and creates Generate operator under Project.
Extracts Generator from the projectList of a Project operator and creates Generate operator under Project.
This rule will throw AnalysisException for following cases: 1. Generator is nested in expressions, e.g.
SELECT explode(list) + 1 FROM tbl2. more than one Generator is found in projectList, e.g.SELECT explode(list), explode(list) FROM tbl3. Generator is found in other operators that are not Project or Generate, e.g.SELECT * FROM tbl SORT BY explode(list)- Definition Classes
- Analyzer
- object ExtractWindowExpressions extends Rule[LogicalPlan]
Extracts WindowExpressions from the projectList of a Project operator and aggregateExpressions of an Aggregate operator and creates individual Window operators for every distinct WindowSpecDefinition.
Extracts WindowExpressions from the projectList of a Project operator and aggregateExpressions of an Aggregate operator and creates individual Window operators for every distinct WindowSpecDefinition.
This rule handles three cases:
- A Project having WindowExpressions in its projectList;
- An Aggregate having WindowExpressions in its aggregateExpressions.
- A Filter->Aggregate pattern representing GROUP BY with a HAVING clause and the Aggregate has WindowExpressions in its aggregateExpressions. Note: If there is a GROUP BY clause in the query, aggregations and corresponding filters (expressions in the HAVING clause) should be evaluated before any WindowExpression. If a query has SELECT DISTINCT, the DISTINCT part should be evaluated after all WindowExpressions.
Note: ResolveLateralColumnAliasReference rule is applied before this rule. To guarantee this order, we make sure this rule applies only when the Project or Aggregate doesn't contain any LATERAL_COLUMN_ALIAS_REFERENCE.
For every case, the transformation works as follows: 1. For a list of Expressions (a projectList or an aggregateExpressions), partitions it two lists of Expressions, one for all WindowExpressions and another for all regular expressions. 2. For all WindowExpressions, groups them based on their WindowSpecDefinitions and WindowFunctionTypes. 3. For every distinct WindowSpecDefinition and WindowFunctionType, creates a Window operator and inserts it into the plan tree.
- Definition Classes
- Analyzer
- object GlobalAggregates extends Rule[LogicalPlan]
Turns projections that contain aggregate expressions into aggregations.
Turns projections that contain aggregate expressions into aggregations.
- Definition Classes
- Analyzer
- object HandleNullInputsForUDF extends Rule[LogicalPlan]
Correctly handle null primitive inputs for UDF by adding extra If expression to do the null check.
Correctly handle null primitive inputs for UDF by adding extra If expression to do the null check. When user defines a UDF with primitive parameters, there is no way to tell if the primitive parameter is null or not, so here we assume the primitive input is null-propagatable and we should return null if the input is null.
- Definition Classes
- Analyzer
- object HandleSpecialCommand extends Rule[LogicalPlan]
A rule to handle special commands that need to be notified when analysis is done.
A rule to handle special commands that need to be notified when analysis is done. This rule should run after all other analysis rules are run.
- Definition Classes
- Analyzer
- object LookupFunctions extends Rule[LogicalPlan]
Checks whether a function identifier referenced by an UnresolvedFunction is defined in the function registry.
Checks whether a function identifier referenced by an UnresolvedFunction is defined in the function registry. Note that this rule doesn't try to resolve the UnresolvedFunction. It only performs simple existence check according to the function identifier to quickly identify undefined functions without triggering relation resolution, which may incur potentially expensive partition/schema discovery process in some cases. In order to avoid duplicate external functions lookup, the external function identifier will store in the local hash set externalFunctionNameSet.
- Definition Classes
- Analyzer
- See also
https://issues.apache.org/jira/browse/SPARK-19737
- object ResolveAggregateFunctions extends Rule[LogicalPlan]
This rule finds aggregate expressions that are not in an aggregate operator.
This rule finds aggregate expressions that are not in an aggregate operator. For example, those in a HAVING clause or ORDER BY clause. These expressions are pushed down to the underlying aggregate operator and then projected away after the original operator.
We need to make sure the expressions all fully resolved before looking for aggregate functions and group by expressions from them.
- Definition Classes
- Analyzer
- object ResolveAliases extends Rule[LogicalPlan]
Replaces UnresolvedAliass with concrete aliases.
Replaces UnresolvedAliass with concrete aliases.
- Definition Classes
- Analyzer
- object ResolveBinaryArithmetic extends Rule[LogicalPlan]
For Add: 1.
For Add: 1. if both side are interval, stays the same; 2. else if one side is date and the other is interval, turns it to DateAddInterval; 3. else if one side is interval, turns it to TimeAdd; 4. else if one side is date, turns it to DateAdd ; 5. else stays the same.
For Subtract: 1. if both side are interval, stays the same; 2. else if the left side is date and the right side is interval, turns it to -r); 3. else if the right side is an interval, turns it to -r); 4. else if one side is timestamp, turns it to SubtractTimestamps; 5. else if the right side is date, turns it to DateDiff/SubtractDates; 6. else if the left side is date, turns it to DateSub; 7. else turns it to stays the same.
For Multiply: 1. If one side is interval, turns it to MultiplyInterval; 2. otherwise, stays the same.
For Divide: 1. If the left side is interval, turns it to DivideInterval; 2. otherwise, stays the same.
- Definition Classes
- Analyzer
- object ResolveDeserializer extends Rule[LogicalPlan]
Replaces UnresolvedDeserializer with the deserialization expression that has been resolved to the given input attributes.
Replaces UnresolvedDeserializer with the deserialization expression that has been resolved to the given input attributes.
- Definition Classes
- Analyzer
- object ResolveEncodersInUDF extends Rule[LogicalPlan]
Resolve the encoders for the UDF by explicitly given the attributes.
Resolve the encoders for the UDF by explicitly given the attributes. We give the attributes explicitly in order to handle the case where the data type of the input value is not the same with the internal schema of the encoder, which could cause data loss. For example, the encoder should not cast the input value to Decimal(38, 18) if the actual data type is Decimal(30, 0).
The resolved encoders then will be used to deserialize the internal row to Scala value.
- Definition Classes
- Analyzer
- object ResolveFieldNameAndPosition extends Rule[LogicalPlan]
Rule to resolve, normalize and rewrite field names based on case sensitivity for commands.
Rule to resolve, normalize and rewrite field names based on case sensitivity for commands.
- Definition Classes
- Analyzer
- object ResolveFunctions extends Rule[LogicalPlan]
Replaces UnresolvedFunctionNames with concrete LogicalPlans.
Replaces UnresolvedFunctionNames with concrete LogicalPlans. Replaces UnresolvedFunctions with concrete Expressions. Replaces UnresolvedGenerators with concrete Expressions. Replaces UnresolvedTableValuedFunctions with concrete LogicalPlans.
- Definition Classes
- Analyzer
- object ResolveGenerate extends Rule[LogicalPlan]
Rewrites table generating expressions that either need one or more of the following in order to be resolved:
Rewrites table generating expressions that either need one or more of the following in order to be resolved:
- concrete attribute references for their output.
- to be relocated from a SELECT clause (i.e. from a Project) into a Generate).
Names for the output Attributes are extracted from Alias or MultiAlias expressions that wrap the Generator.
- Definition Classes
- Analyzer
- object ResolveGroupingAnalytics extends Rule[LogicalPlan]
- Definition Classes
- Analyzer
- object ResolveInsertInto extends ResolveInsertionBase
Handle INSERT INTO for DSv2
Handle INSERT INTO for DSv2
- Definition Classes
- Analyzer
- object ResolveNaturalAndUsingJoin extends Rule[LogicalPlan]
Removes natural or using joins by calculating output columns based on output from two sides, Then apply a Project on a normal Join to eliminate natural or using join.
Removes natural or using joins by calculating output columns based on output from two sides, Then apply a Project on a normal Join to eliminate natural or using join.
- Definition Classes
- Analyzer
- object ResolveNewInstance extends Rule[LogicalPlan]
Resolves NewInstance by finding and adding the outer scope to it if the object being constructed is an inner class.
Resolves NewInstance by finding and adding the outer scope to it if the object being constructed is an inner class.
- Definition Classes
- Analyzer
- object ResolveOrdinalInOrderByAndGroupBy extends Rule[LogicalPlan]
In many dialects of SQL it is valid to use ordinal positions in order/sort by and group by clauses.
In many dialects of SQL it is valid to use ordinal positions in order/sort by and group by clauses. This rule is to convert ordinal positions to the corresponding expressions in the select list. This support is introduced in Spark 2.0.
- When the sort references or group by expressions are not integer but foldable expressions, just ignore them. - When spark.sql.orderByOrdinal/spark.sql.groupByOrdinal is set to false, ignore the position numbers too.
Before the release of Spark 2.0, the literals in order/sort by and group by clauses have no effect on the results.
- Definition Classes
- Analyzer
- object ResolveOutputRelation extends Rule[LogicalPlan]
Resolves columns of an output table from the data in a logical plan.
Resolves columns of an output table from the data in a logical plan. This rule will:
- Reorder columns when the write is by name - Insert casts when data types do not match - Insert aliases when column names do not match - Detect plans that are not compatible with the output table and throw AnalysisException
- Definition Classes
- Analyzer
- object ResolvePivot extends Rule[LogicalPlan]
- Definition Classes
- Analyzer
- object ResolveRandomSeed extends Rule[LogicalPlan]
Set the seed for random number generation.
Set the seed for random number generation.
- Definition Classes
- Analyzer
- object ResolveReferences extends Rule[LogicalPlan] with ColumnResolutionHelper
Resolves column references in the query plan.
Resolves column references in the query plan. Basically it transform the query plan tree bottom up, and only try to resolve references for a plan node if all its children nodes are resolved, and there is no conflicting attributes between the children nodes (see
hasConflictingAttrsfor details).The general workflow to resolve references: 1. Expands the star in Project/Aggregate/Generate. 2. Resolves the columns to AttributeReference with the output of the children plans. This includes metadata columns as well. 3. Resolves the columns to literal function which is allowed to be invoked without braces, e.g.
SELECT col, current_date FROM t. 4. Resolves the columns to outer references with the outer plan if we are resolving subquery expressions.Some plan nodes have special column reference resolution logic, please read these sub-rules for details:
Note: even if we use a single rule to resolve columns, it's still non-trivial to have a reliable column resolution order, as the rule will be executed multiple times, with other rules in the same batch. We should resolve columns with the next option only if all the previous options are permanently not applicable. If the current option can be applicable in the next iteration (other rules update the plan), we should not try the next option.
- Definition Classes
- Analyzer
- object ResolveRelations extends Rule[LogicalPlan]
Replaces unresolved relations (tables and views) with concrete relations from the catalog.
Replaces unresolved relations (tables and views) with concrete relations from the catalog.
- Definition Classes
- Analyzer
- object ResolveSubquery extends Rule[LogicalPlan]
This rule resolves and rewrites subqueries inside expressions.
This rule resolves and rewrites subqueries inside expressions.
Note: CTEs are handled in CTESubstitution.
- Definition Classes
- Analyzer
- object ResolveSubqueryColumnAliases extends Rule[LogicalPlan]
Replaces unresolved column aliases for a subquery with projections.
Replaces unresolved column aliases for a subquery with projections.
- Definition Classes
- Analyzer
- object ResolveUnpivot extends Rule[LogicalPlan]
- Definition Classes
- Analyzer
- object ResolveUpCast extends Rule[LogicalPlan]
Replace the UpCast expression by Cast, and throw exceptions if the cast may truncate.
Replace the UpCast expression by Cast, and throw exceptions if the cast may truncate.
- Definition Classes
- Analyzer
- object ResolveWindowFrame extends Rule[LogicalPlan]
Check and add proper window frames for all window functions.
Check and add proper window frames for all window functions.
- Definition Classes
- Analyzer
- object ResolveWindowOrder extends Rule[LogicalPlan]
Check and add order to AggregateWindowFunctions.
Check and add order to AggregateWindowFunctions.
- Definition Classes
- Analyzer
- object WindowsSubstitution extends Rule[LogicalPlan]
Substitute child plan with WindowSpecDefinitions.
Substitute child plan with WindowSpecDefinitions.
- Definition Classes
- Analyzer
- case object Once extends Strategy with Product with Serializable
A strategy that is run once and idempotent.
A strategy that is run once and idempotent.
- Definition Classes
- RuleExecutor
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- val DATA_TYPE_MISMATCH_ERROR: TreeNodeTag[Boolean]
- Definition Classes
- CheckAnalysis
- val INVALID_FORMAT_ERROR: TreeNodeTag[Boolean]
- Definition Classes
- CheckAnalysis
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def batches: Seq[Batch]
Defines a sequence of rule batches, to be overridden by the implementation.
Defines a sequence of rule batches, to be overridden by the implementation.
- Definition Classes
- Analyzer → RuleExecutor
- def buildBalancedPredicate(expressions: Seq[Expression], op: (Expression, Expression) => Expression): Expression
Builds a balanced output predicate in bottom up approach, by applying binary operator op pair by pair on input predicates exprs recursively.
Builds a balanced output predicate in bottom up approach, by applying binary operator op pair by pair on input predicates exprs recursively. Example: exprs = [a, b, c, d], op = And, returns (a And b) And (c And d) exprs = [a, b, c, d, e, f], op = And, returns ((a And b) And (c And d)) And (e And f)
- Attributes
- protected
- Definition Classes
- PredicateHelper
- def canEvaluate(expr: Expression, plan: LogicalPlan): Boolean
Returns true if
exprcan be evaluated using only the output ofplan.Returns true if
exprcan be evaluated using only the output ofplan. This method can be used to determine when it is acceptable to move expression evaluation within a query plan.For example consider a join between two relations R(a, b) and S(c, d).
-
canEvaluate(EqualTo(a,b), R)returnstrue-canEvaluate(EqualTo(a,c), R)returnsfalse-canEvaluate(Literal(1), R)returnstrueas literals CAN be evaluated on any plan- Attributes
- protected
- Definition Classes
- PredicateHelper
- def canEvaluateWithinJoin(expr: Expression): Boolean
Returns true iff
exprcould be evaluated as a condition within join.Returns true iff
exprcould be evaluated as a condition within join.- Attributes
- protected
- Definition Classes
- PredicateHelper
- val catalogManager: CatalogManager
- Definition Classes
- Analyzer → LookupCatalog
- def checkAnalysis(plan: LogicalPlan): Unit
- Definition Classes
- CheckAnalysis
- def checkAnalysis0(plan: LogicalPlan): Unit
- Definition Classes
- CheckAnalysis
- def checkSubqueryExpression(plan: LogicalPlan, expr: SubqueryExpression): Unit
Validates subquery expressions in the plan.
Validates subquery expressions in the plan. Upon failure, returns an user facing error.
- Definition Classes
- CheckAnalysis
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @native()
- def conf: SQLConf
The active config object within the current scope.
The active config object within the current scope. See SQLConf.get for more information.
- Definition Classes
- SQLConfHelper
- def currentCatalog: CatalogPlugin
Returns the current catalog set.
Returns the current catalog set.
- Definition Classes
- LookupCatalog
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- val excludedOnceBatches: Set[String]
Once batches that are excluded in the idempotence checker
Once batches that are excluded in the idempotence checker
- Attributes
- protected
- Definition Classes
- RuleExecutor
- def execute(plan: LogicalPlan): LogicalPlan
Executes the batches of rules defined by the subclass.
Executes the batches of rules defined by the subclass. The batches are executed serially using the defined execution strategy. Within each batch, rules are also executed serially.
- Definition Classes
- Analyzer → RuleExecutor
- def executeAndCheck(plan: LogicalPlan, tracker: QueryPlanningTracker): LogicalPlan
- Definition Classes
- Analyzer
- def executeAndTrack(plan: LogicalPlan, tracker: QueryPlanningTracker): LogicalPlan
Executes the batches of rules defined by the subclass, and also tracks timing info for each rule using the provided tracker.
Executes the batches of rules defined by the subclass, and also tracks timing info for each rule using the provided tracker.
- Definition Classes
- RuleExecutor
- See also
- val extendedCheckRules: Seq[(LogicalPlan) => Unit]
Override to provide additional checks for correct analysis.
Override to provide additional checks for correct analysis. These rules will be evaluated after our built-in check rules.
- Definition Classes
- CheckAnalysis
- val extendedResolutionRules: Seq[Rule[LogicalPlan]]
Override to provide additional rules for the "Resolution" batch.
Override to provide additional rules for the "Resolution" batch.
- Definition Classes
- Analyzer
- def extractPredicatesWithinOutputSet(condition: Expression, outputSet: AttributeSet): Option[Expression]
Returns a filter that its reference is a subset of
outputSetand it contains the maximum constraints fromcondition.Returns a filter that its reference is a subset of
outputSetand it contains the maximum constraints fromcondition. This is used for predicate pushdown. When there is no such filter,Noneis returned.- Attributes
- protected
- Definition Classes
- PredicateHelper
- def failAnalysis(errorClass: String, messageParameters: Map[String, String]): Nothing
Fails the analysis at the point where a specific tree node was parsed using a provided error class and message parameters.
Fails the analysis at the point where a specific tree node was parsed using a provided error class and message parameters.
- Definition Classes
- CheckAnalysis
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable])
- def findExpressionAndTrackLineageDown(exp: Expression, plan: LogicalPlan): Option[(Expression, LogicalPlan)]
Find the origin of where the input references of expression exp were scanned in the tree of plan, and if they originate from a single leaf node.
Find the origin of where the input references of expression exp were scanned in the tree of plan, and if they originate from a single leaf node. Returns optional tuple with Expression, undoing any projections and aliasing that has been done along the way from plan to origin, and the origin LeafNode plan from which all the exp
- Definition Classes
- PredicateHelper
- def fixedPoint: FixedPoint
If the plan cannot be resolved within maxIterations, analyzer will throw exception to inform user to increase the value of SQLConf.ANALYZER_MAX_ITERATIONS.
If the plan cannot be resolved within maxIterations, analyzer will throw exception to inform user to increase the value of SQLConf.ANALYZER_MAX_ITERATIONS.
- Attributes
- protected
- Definition Classes
- Analyzer
- def getAliasMap(exprs: Seq[NamedExpression]): AttributeMap[Alias]
- Attributes
- protected
- Definition Classes
- AliasHelper
- def getAliasMap(plan: Aggregate): AttributeMap[Alias]
- Attributes
- protected
- Definition Classes
- AliasHelper
- def getAliasMap(plan: Project): AttributeMap[Alias]
- Attributes
- protected
- Definition Classes
- AliasHelper
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def getQueryContext(sqlContext: SQLQueryContext): Array[QueryContext]
- Definition Classes
- DataTypeErrorsBase
- def getSummary(sqlContext: SQLQueryContext): String
- Definition Classes
- DataTypeErrorsBase
- def hasMapType(dt: DataType): Boolean
- Attributes
- protected
- Definition Classes
- CheckAnalysis
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
- Attributes
- protected
- Definition Classes
- Logging
- def initializeLogIfNecessary(isInterpreter: Boolean): Unit
- Attributes
- protected
- Definition Classes
- Logging
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- def isLikelySelective(e: Expression): Boolean
Returns whether an expression is likely to be selective
Returns whether an expression is likely to be selective
- Definition Classes
- PredicateHelper
- def isNullIntolerant(expr: Expression): Boolean
- Attributes
- protected
- Definition Classes
- PredicateHelper
- def isTraceEnabled(): Boolean
- Attributes
- protected
- Definition Classes
- Logging
- def isView(nameParts: Seq[String]): Boolean
- Definition Classes
- Analyzer → CheckAnalysis
- def log: Logger
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logName: String
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def mapColumnInSetOperation(plan: LogicalPlan): Option[Attribute]
- Attributes
- protected
- Definition Classes
- CheckAnalysis
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- def outputWithNullability(output: Seq[Attribute], nonNullAttrExprIds: Seq[ExprId]): Seq[Attribute]
- Attributes
- protected
- Definition Classes
- PredicateHelper
- val postHocResolutionRules: Seq[Rule[LogicalPlan]]
Override to provide rules to do post-hoc resolution.
Override to provide rules to do post-hoc resolution. Note that these rules will be executed in an individual batch. This batch is to run right after the normal resolution batch and execute its rules in one pass.
- Definition Classes
- Analyzer
- def quoteByDefault(elem: String): String
- Attributes
- protected
- Definition Classes
- DataTypeErrorsBase
- def replaceAlias(expr: Expression, aliasMap: AttributeMap[Alias]): Expression
Replace all attributes, that reference an alias, with the aliased expression
Replace all attributes, that reference an alias, with the aliased expression
- Attributes
- protected
- Definition Classes
- AliasHelper
- def replaceAliasButKeepName(expr: NamedExpression, aliasMap: AttributeMap[Alias]): NamedExpression
Replace all attributes, that reference an alias, with the aliased expression, but keep the name of the outermost attribute.
Replace all attributes, that reference an alias, with the aliased expression, but keep the name of the outermost attribute.
- Attributes
- protected
- Definition Classes
- AliasHelper
- def resolveColWithAgg(e: Expression, plan: LogicalPlan): Expression
- Attributes
- protected
- Definition Classes
- ColumnResolutionHelper
- def resolveExprInAssignment(expr: Expression, hostPlan: LogicalPlan): Expression
- Definition Classes
- ColumnResolutionHelper
- def resolveExpressionByPlanChildren(e: Expression, q: LogicalPlan, allowOuter: Boolean = false): Expression
Resolves
UnresolvedAttribute,GetColumnByOrdinaland extract value expressions(s) by the input plan's children output attributes.Resolves
UnresolvedAttribute,GetColumnByOrdinaland extract value expressions(s) by the input plan's children output attributes.- e
The expression need to be resolved.
- q
The LogicalPlan whose children are used to resolve expression's attribute.
- returns
resolved Expression.
- Definition Classes
- ColumnResolutionHelper
- def resolveExpressionByPlanOutput(expr: Expression, plan: LogicalPlan, throws: Boolean = false, allowOuter: Boolean = false): Expression
Resolves
UnresolvedAttribute,GetColumnByOrdinaland extract value expressions(s) by the input plan's output attributes.Resolves
UnresolvedAttribute,GetColumnByOrdinaland extract value expressions(s) by the input plan's output attributes. In order to resolve the nested fields correctly, this function makes use ofthrowsparameter to control when to raise an AnalysisException.Example : SELECT * FROM t ORDER BY a.b
In the above example, after
ais resolved to a struct-type column, we may fail to resolvebif there is no such nested field named "b". We should not fail and wait for other rules to resolve it if possible.- Definition Classes
- ColumnResolutionHelper
- def resolveExprsAndAddMissingAttrs(exprs: Seq[Expression], plan: LogicalPlan): (Seq[Expression], LogicalPlan)
This method tries to resolve expressions and find missing attributes recursively.
This method tries to resolve expressions and find missing attributes recursively. Specifically, when the expressions used in
SortorFiltercontain unresolved attributes or resolved attributes which are missing from child output. This method tries to find the missing attributes and add them into the projection.- Attributes
- protected
- Definition Classes
- ColumnResolutionHelper
- def resolveLateralColumnAlias(selectList: Seq[Expression]): Seq[Expression]
- Attributes
- protected
- Definition Classes
- ColumnResolutionHelper
- def resolveOuterRef(e: Expression): Expression
- Attributes
- protected
- Definition Classes
- ColumnResolutionHelper
- def resolver: Resolver
- Definition Classes
- Analyzer
- def splitConjunctivePredicates(condition: Expression): Seq[Expression]
- Attributes
- protected
- Definition Classes
- PredicateHelper
- def splitDisjunctivePredicates(condition: Expression): Seq[Expression]
- Attributes
- protected
- Definition Classes
- PredicateHelper
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toDSOption(option: String): String
- Definition Classes
- QueryErrorsBase
- def toSQLConf(conf: String): String
- Definition Classes
- DataTypeErrorsBase
- def toSQLConfVal(conf: String): String
- Definition Classes
- QueryErrorsBase
- def toSQLExpr(e: Expression): String
- Definition Classes
- QueryErrorsBase
- def toSQLId(parts: Seq[String]): String
- Definition Classes
- DataTypeErrorsBase
- def toSQLId(parts: String): String
- Definition Classes
- DataTypeErrorsBase
- def toSQLSchema(schema: String): String
- Definition Classes
- QueryErrorsBase
- def toSQLStmt(text: String): String
- Definition Classes
- DataTypeErrorsBase
- def toSQLType(t: AbstractDataType): String
- Definition Classes
- DataTypeErrorsBase
- def toSQLType(text: String): String
- Definition Classes
- DataTypeErrorsBase
- def toSQLValue(v: Any, t: DataType): String
- Definition Classes
- QueryErrorsBase
- def toSQLValue(value: Double): String
- Definition Classes
- DataTypeErrorsBase
- def toSQLValue(value: Float): String
- Definition Classes
- DataTypeErrorsBase
- def toSQLValue(value: Long): String
- Definition Classes
- DataTypeErrorsBase
- def toSQLValue(value: Int): String
- Definition Classes
- DataTypeErrorsBase
- def toSQLValue(value: Short): String
- Definition Classes
- DataTypeErrorsBase
- def toSQLValue(value: UTF8String): String
- Definition Classes
- DataTypeErrorsBase
- def toSQLValue(value: String): String
- Definition Classes
- DataTypeErrorsBase
- def toString(): String
- Definition Classes
- AnyRef → Any
- def trimAliases(e: Expression): Expression
- Attributes
- protected
- Definition Classes
- AliasHelper
- def trimNonTopLevelAliases[T <: Expression](e: T): T
- Attributes
- protected
- Definition Classes
- AliasHelper
- def validatePlanChanges(previousPlan: LogicalPlan, currentPlan: LogicalPlan): Option[String]
Defines a validate function that validates the plan changes after the execution of each rule, to make sure these rules make valid changes to the plan.
Defines a validate function that validates the plan changes after the execution of each rule, to make sure these rules make valid changes to the plan. For example, we can check whether a plan is still resolved after each rule in
Optimizer, so that we can catch rules that turn the plan into unresolved.- Attributes
- protected
- Definition Classes
- Analyzer → RuleExecutor
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- object AsTableIdentifier
Extract legacy table identifier from a multi-part identifier.
Extract legacy table identifier from a multi-part identifier.
For legacy support only. Please use CatalogAndIdentifier instead on DSv2 code paths.
- Definition Classes
- LookupCatalog
- object CatalogAndIdentifier
Extract catalog and identifier from a multi-part name with the current catalog if needed.
Extract catalog and identifier from a multi-part name with the current catalog if needed. Catalog name takes precedence over identifier, but for a single-part name, identifier takes precedence over catalog name.
Note that, this pattern is used to look up permanent catalog objects like table, view, function, etc. If you need to look up temp objects like temp view, please do it separately before calling this pattern, as temp objects don't belong to any catalog.
- Definition Classes
- LookupCatalog
- object CatalogAndNamespace
Extract catalog and namespace from a multi-part name with the current catalog if needed.
Extract catalog and namespace from a multi-part name with the current catalog if needed. Catalog name takes precedence over namespaces.
- Definition Classes
- LookupCatalog
- object NonSessionCatalogAndIdentifier
Extract non-session catalog and identifier from a multi-part identifier.
Extract non-session catalog and identifier from a multi-part identifier.
- Definition Classes
- LookupCatalog
- object SessionCatalogAndIdentifier
Extract session catalog and identifier from a multi-part identifier.
Extract session catalog and identifier from a multi-part identifier.
- Definition Classes
- LookupCatalog