org.apache.hadoop.hive.ql.optimizer.calcite

Class HiveCalciteUtil

java.lang.Object

org.apache.hadoop.hive.ql.optimizer.calcite.HiveCalciteUtil

public class HiveCalciteUtil
extends Object

Generic utility functions needed for Calcite based Hive CBO.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HiveCalciteUtil.ConstantFinder Walks over an expression and determines whether it is constant.
static class	HiveCalciteUtil.FixNullabilityShuttle Shuttle that fixes up an expression to match changes in nullability of input fields.
static class	HiveCalciteUtil.JoinLeafPredicateInfo JoinLeafPredicateInfo represents leaf predicate in Join condition (conjuctive lement).
static class	HiveCalciteUtil.JoinPredicateInfo JoinPredicateInfo represents Join condition; JoinPredicate Info uses JoinLeafPredicateInfo to represent individual conjunctive elements in the predicate. JoinPredicateInfo = JoinLeafPredicateInfo1 and JoinLeafPredicateInfo2...

Field Summary

Fields

Modifier and Type	Field and Description
static com.google.common.base.Function<org.apache.calcite.rex.RexNode,String>	REX_STR_FN

Constructor Summary

Constructors

Constructor and Description
HiveCalciteUtil()

Method Summary

Modifier and Type	Method and Description
static org.apache.calcite.rex.RexCall	checkMaterializable(org.apache.calcite.rex.RexNode expr) Check if the expression is usable for query materialization, returning the first failing expression.
static HiveProject	createProjectWithoutColumn(org.apache.calcite.rel.RelNode input, Set<Integer> positions)
static org.apache.calcite.rel.core.AggregateCall	createSingleArgAggCall(String funcName, org.apache.calcite.plan.RelOptCluster cluster, PrimitiveTypeInfo typeInfo, Integer pos, org.apache.calcite.rel.type.RelDataType aggFnRetType)
static HiveTableFunctionScan	createUDTFForSetOp(org.apache.calcite.plan.RelOptCluster cluster, org.apache.calcite.rel.RelNode input)
static org.apache.calcite.util.ImmutableBitSet	extractRefs(org.apache.calcite.rel.core.Aggregate aggregate) Extracts inputs referenced by aggregate operator.
static Set<org.apache.calcite.rex.RexTableInputRef>	findRexTableInputRefs(org.apache.calcite.rex.RexNode rexNode)
static List<org.apache.calcite.rex.RexNode>	fixNullability(org.apache.calcite.rex.RexBuilder rexBuilder, List<org.apache.calcite.rex.RexNode> nodes, List<org.apache.calcite.rel.type.RelDataType> fieldTypes) Fixes up the type of all RexInputRefs in an expression to match differences in nullability.
static org.apache.calcite.rex.RexNode	fixNullability(org.apache.calcite.rex.RexBuilder rexBuilder, org.apache.calcite.rex.RexNode node, List<org.apache.calcite.rel.type.RelDataType> fieldTypes) Fixes up the type of all RexInputRefs in an expression to match differences in nullability.
static <T> com.google.common.collect.ImmutableMap<Integer,T>	getColInfoMap(List<T> hiveCols, int startIndx)
static com.google.common.collect.ImmutableMap<String,Integer>	getColNameIndxMap(List<org.apache.hadoop.hive.metastore.api.FieldSchema> tableFields)
List<org.apache.calcite.rex.RexNode>	getDeterministicFuncWithSingleInputRef(List<org.apache.calcite.rex.RexNode> exprs, Set<Integer> validInputRefs)
static ExprNodeDesc	getExprNode(Integer inputRefIndx, org.apache.calcite.rel.RelNode inputRel, ExprNodeConverter exprConv)
static List<ExprNodeDesc>	getExprNodes(List<Integer> inputRefs, org.apache.calcite.rel.RelNode inputRel, String inputTabAlias)
static List<String>	getFieldNames(List<Integer> inputRefs, org.apache.calcite.rel.RelNode inputRel)
static com.google.common.collect.ImmutableList<org.apache.calcite.rex.RexNode>	getInputRef(List<Integer> inputRefs, org.apache.calcite.rel.RelNode inputRel)
static Set<Integer>	getInputRefs(org.apache.calcite.rex.RexNode expr)
static com.google.common.collect.ImmutableList<org.apache.calcite.rex.RexNode>	getPredsNotPushedAlready(org.apache.calcite.rel.RelNode inp, List<org.apache.calcite.rex.RexNode> predsToPushDown)
static com.google.common.collect.ImmutableList<org.apache.calcite.rex.RexNode>	getPredsNotPushedAlready(Set<String> predicatesToExclude, org.apache.calcite.rel.RelNode inp, List<org.apache.calcite.rex.RexNode> predsToPushDown) Given a list of predicates to push down, this methods returns the set of predicates that still need to be pushed.
static List<org.apache.calcite.rex.RexNode>	getProjsFromBelowAsInputRef(org.apache.calcite.rel.RelNode rel)
static com.google.common.collect.ImmutableMap<String,Integer>	getRowColNameIndxMap(List<org.apache.calcite.rel.type.RelDataTypeField> rowFields)
static org.apache.calcite.util.Pair<org.apache.calcite.rel.RelNode,org.apache.calcite.rel.RelNode>	getTopLevelSelect(org.apache.calcite.rel.RelNode rootRel) Get top level select starting from root.
static org.apache.calcite.rex.RexNode	getTypeSafePred(org.apache.calcite.plan.RelOptCluster cluster, org.apache.calcite.rex.RexNode rex, org.apache.calcite.rel.type.RelDataType rType)
static com.google.common.collect.ImmutableMap<Integer,VirtualColumn>	getVColsMap(List<VirtualColumn> hiveVCols, int startIndx)
static boolean	hasAllExpressionsFromRightSide(org.apache.calcite.rel.RelNode joinRel, List<org.apache.calcite.rex.RexNode> expressions)
static boolean	hasAnyExpressionFromRightSide(org.apache.calcite.rel.RelNode joinRel, List<org.apache.calcite.rex.RexNode> expressions) Checks if any of the expression given as list expressions are from right side of the join.
static boolean	hasDisjuction(org.apache.calcite.rex.RexNode node) Returns whether the expression has disjunctions (OR) at any level of nesting.
static boolean	isComparisonOp(org.apache.calcite.rex.RexCall call)
static boolean	isDeterministic(org.apache.calcite.rex.RexNode expr)
static boolean	isDeterministicFuncOnLiterals(org.apache.calcite.rex.RexNode expr)
static boolean	isDeterministicFuncWithSingleInputRef(org.apache.calcite.rex.RexNode expr, Set<Integer> validInputRefs)
static boolean	isLiteral(org.apache.calcite.rex.RexNode expr)
static boolean	isMaterializable(org.apache.calcite.rex.RexNode expr) Is the expression usable for query materialization.
static boolean	limitRelNode(org.apache.calcite.rel.RelNode rel)
static boolean	orderRelNode(org.apache.calcite.rel.RelNode rel)
static org.apache.calcite.rex.RexNode	projectNonColumnEquiConditions(org.apache.calcite.rel.core.RelFactories.ProjectFactory factory, org.apache.calcite.rel.RelNode[] inputRels, List<org.apache.calcite.rex.RexNode> leftJoinKeys, List<org.apache.calcite.rex.RexNode> rightJoinKeys, int systemColCount, List<Integer> leftKeys, List<Integer> rightKeys) Push any equi join conditions that are not column references as Projections on top of the children.
static boolean	pureLimitRelNode(org.apache.calcite.rel.RelNode rel)
static boolean	pureOrderRelNode(org.apache.calcite.rel.RelNode rel)
static com.google.common.collect.ImmutableSet<Integer>	shiftVColsSet(Set<Integer> hiveVCols, int shift)
static List<Integer>	translateBitSetToProjIndx(org.apache.calcite.util.ImmutableBitSet projBitSet)
static boolean	validateASTForUnsupportedTokens(ASTNode ast)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

REX_STR_FN

public static final com.google.common.base.Function<org.apache.calcite.rex.RexNode,String> REX_STR_FN

Constructor Detail

HiveCalciteUtil

public HiveCalciteUtil()

Method Detail

validateASTForUnsupportedTokens

public static boolean validateASTForUnsupportedTokens(ASTNode ast)

getProjsFromBelowAsInputRef

public static List<org.apache.calcite.rex.RexNode> getProjsFromBelowAsInputRef(org.apache.calcite.rel.RelNode rel)

translateBitSetToProjIndx

public static List<Integer> translateBitSetToProjIndx(org.apache.calcite.util.ImmutableBitSet projBitSet)

projectNonColumnEquiConditions

public static org.apache.calcite.rex.RexNode projectNonColumnEquiConditions(org.apache.calcite.rel.core.RelFactories.ProjectFactory factory,
                                                                            org.apache.calcite.rel.RelNode[] inputRels,
                                                                            List<org.apache.calcite.rex.RexNode> leftJoinKeys,
                                                                            List<org.apache.calcite.rex.RexNode> rightJoinKeys,
                                                                            int systemColCount,
                                                                            List<Integer> leftKeys,
                                                                            List<Integer> rightKeys)

Push any equi join conditions that are not column references as Projections on top of the children.

Parameters:

factory - Project factory to use.

inputRels - inputs to a join

leftJoinKeys - expressions for LHS of join key

rightJoinKeys - expressions for RHS of join key

systemColCount - number of system columns, usually zero. These columns are projected at the leading edge of the output row.

leftKeys - on return this contains the join key positions from the new project rel on the LHS.

rightKeys - on return this contains the join key positions from the new project rel on the RHS.

Returns:

the join condition after the equi expressions pushed down.

pureLimitRelNode

public static boolean pureLimitRelNode(org.apache.calcite.rel.RelNode rel)

pureOrderRelNode

public static boolean pureOrderRelNode(org.apache.calcite.rel.RelNode rel)

limitRelNode

public static boolean limitRelNode(org.apache.calcite.rel.RelNode rel)

orderRelNode

public static boolean orderRelNode(org.apache.calcite.rel.RelNode rel)

getTopLevelSelect

public static org.apache.calcite.util.Pair<org.apache.calcite.rel.RelNode,org.apache.calcite.rel.RelNode> getTopLevelSelect(org.apache.calcite.rel.RelNode rootRel)

Get top level select starting from root. Assumption here is root can only be Sort & Project. Also the top project should be at most 2 levels below Sort; i.e Sort(Limit)-Sort(OB)-Select

Parameters:

rootRel -

Returns:

isComparisonOp

public static boolean isComparisonOp(org.apache.calcite.rex.RexCall call)

getPredsNotPushedAlready

public static com.google.common.collect.ImmutableList<org.apache.calcite.rex.RexNode> getPredsNotPushedAlready(org.apache.calcite.rel.RelNode inp,
                                                                                                               List<org.apache.calcite.rex.RexNode> predsToPushDown)

getPredsNotPushedAlready

public static com.google.common.collect.ImmutableList<org.apache.calcite.rex.RexNode> getPredsNotPushedAlready(Set<String> predicatesToExclude,
                                                                                                               org.apache.calcite.rel.RelNode inp,
                                                                                                               List<org.apache.calcite.rex.RexNode> predsToPushDown)

Given a list of predicates to push down, this methods returns the set of predicates that still need to be pushed. Predicates need to be pushed because 1) their String representation is not included in input set of predicates to exclude, or 2) they are already in the subtree rooted at the input node. This method updates the set of predicates to exclude with the String representation of the predicates in the output and in the subtree.

Parameters:

predicatesToExclude - String representation of predicates that should be excluded

inp - root of the subtree

predsToPushDown - candidate predicates to push down through the subtree

Returns:

list of predicates to push down

getTypeSafePred

public static org.apache.calcite.rex.RexNode getTypeSafePred(org.apache.calcite.plan.RelOptCluster cluster,
                                                             org.apache.calcite.rex.RexNode rex,
                                                             org.apache.calcite.rel.type.RelDataType rType)

isDeterministic

public static boolean isDeterministic(org.apache.calcite.rex.RexNode expr)

isDeterministicFuncOnLiterals

public static boolean isDeterministicFuncOnLiterals(org.apache.calcite.rex.RexNode expr)

getDeterministicFuncWithSingleInputRef

public List<org.apache.calcite.rex.RexNode> getDeterministicFuncWithSingleInputRef(List<org.apache.calcite.rex.RexNode> exprs,
                                                                                   Set<Integer> validInputRefs)

isDeterministicFuncWithSingleInputRef

public static boolean isDeterministicFuncWithSingleInputRef(org.apache.calcite.rex.RexNode expr,
                                                            Set<Integer> validInputRefs)

getColInfoMap

public static <T> com.google.common.collect.ImmutableMap<Integer,T> getColInfoMap(List<T> hiveCols,
                                                                                  int startIndx)

shiftVColsSet

public static com.google.common.collect.ImmutableSet<Integer> shiftVColsSet(Set<Integer> hiveVCols,
                                                                            int shift)

getVColsMap

public static com.google.common.collect.ImmutableMap<Integer,VirtualColumn> getVColsMap(List<VirtualColumn> hiveVCols,
                                                                                        int startIndx)

getColNameIndxMap

public static com.google.common.collect.ImmutableMap<String,Integer> getColNameIndxMap(List<org.apache.hadoop.hive.metastore.api.FieldSchema> tableFields)

getRowColNameIndxMap

public static com.google.common.collect.ImmutableMap<String,Integer> getRowColNameIndxMap(List<org.apache.calcite.rel.type.RelDataTypeField> rowFields)

getInputRef

public static com.google.common.collect.ImmutableList<org.apache.calcite.rex.RexNode> getInputRef(List<Integer> inputRefs,
                                                                                                  org.apache.calcite.rel.RelNode inputRel)

getExprNode

public static ExprNodeDesc getExprNode(Integer inputRefIndx,
                                       org.apache.calcite.rel.RelNode inputRel,
                                       ExprNodeConverter exprConv)

getExprNodes

public static List<ExprNodeDesc> getExprNodes(List<Integer> inputRefs,
                                              org.apache.calcite.rel.RelNode inputRel,
                                              String inputTabAlias)

getFieldNames

public static List<String> getFieldNames(List<Integer> inputRefs,
                                         org.apache.calcite.rel.RelNode inputRel)

createSingleArgAggCall

public static org.apache.calcite.rel.core.AggregateCall createSingleArgAggCall(String funcName,
                                                                               org.apache.calcite.plan.RelOptCluster cluster,
                                                                               PrimitiveTypeInfo typeInfo,
                                                                               Integer pos,
                                                                               org.apache.calcite.rel.type.RelDataType aggFnRetType)

isMaterializable

public static boolean isMaterializable(org.apache.calcite.rex.RexNode expr)

Is the expression usable for query materialization.

checkMaterializable

public static org.apache.calcite.rex.RexCall checkMaterializable(org.apache.calcite.rex.RexNode expr)

Check if the expression is usable for query materialization, returning the first failing expression.

createUDTFForSetOp

public static HiveTableFunctionScan createUDTFForSetOp(org.apache.calcite.plan.RelOptCluster cluster,
                                                       org.apache.calcite.rel.RelNode input)
                                                throws SemanticException

Throws:

SemanticException

createProjectWithoutColumn

public static HiveProject createProjectWithoutColumn(org.apache.calcite.rel.RelNode input,
                                                     Set<Integer> positions)
                                              throws CalciteSemanticException

Throws:

CalciteSemanticException

isLiteral

public static boolean isLiteral(org.apache.calcite.rex.RexNode expr)

getInputRefs

public static Set<Integer> getInputRefs(org.apache.calcite.rex.RexNode expr)

fixNullability

public static List<org.apache.calcite.rex.RexNode> fixNullability(org.apache.calcite.rex.RexBuilder rexBuilder,
                                                                  List<org.apache.calcite.rex.RexNode> nodes,
                                                                  List<org.apache.calcite.rel.type.RelDataType> fieldTypes)

Fixes up the type of all RexInputRefs in an expression to match differences in nullability. This can be useful in case a field is inferred to be not nullable, e.g., a not null literal, and the reference to the row type needs to be changed to adjust the nullability flag. In case of references created on top of a Calcite schema generated directly from a Hive schema, this is especially useful since Hive does not have a notion of nullability so all fields in the schema will be inferred to nullable. However, Calcite makes this distinction.

Throws if there any greater inconsistencies of type.

fixNullability

public static org.apache.calcite.rex.RexNode fixNullability(org.apache.calcite.rex.RexBuilder rexBuilder,
                                                            org.apache.calcite.rex.RexNode node,
                                                            List<org.apache.calcite.rel.type.RelDataType> fieldTypes)

Fixes up the type of all RexInputRefs in an expression to match differences in nullability.

Throws if there any greater inconsistencies of type.

hasDisjuction

public static boolean hasDisjuction(org.apache.calcite.rex.RexNode node)

Returns whether the expression has disjunctions (OR) at any level of nesting.

• Example 1: OR(=($0, $1), IS NOT NULL($2))):INTEGER (OR in the top-level expression)
• Example 2: NOT(AND(=($0, $1), IS NOT NULL($2)) this is equivalent to OR((<>($0, $1), IS NULL($2))
• Example 3: AND(OR(=($0, $1), IS NOT NULL($2)))) (OR in inner expression)

Parameters:

node - the expression where to look for disjunctions.

Returns:

true if the given expressions contains a disjunction, false otherwise.

hasAnyExpressionFromRightSide

public static boolean hasAnyExpressionFromRightSide(org.apache.calcite.rel.RelNode joinRel,
                                                    List<org.apache.calcite.rex.RexNode> expressions)

Checks if any of the expression given as list expressions are from right side of the join. This is used during anti join conversion.

Parameters:

joinRel - Join node whose right side has to be searched.

expressions - The list of expression to search.

Returns:

true if any of the expressions is from right side of join.

hasAllExpressionsFromRightSide

public static boolean hasAllExpressionsFromRightSide(org.apache.calcite.rel.RelNode joinRel,
                                                     List<org.apache.calcite.rex.RexNode> expressions)

extractRefs

public static org.apache.calcite.util.ImmutableBitSet extractRefs(org.apache.calcite.rel.core.Aggregate aggregate)

Extracts inputs referenced by aggregate operator.

findRexTableInputRefs

public static Set<org.apache.calcite.rex.RexTableInputRef> findRexTableInputRefs(org.apache.calcite.rex.RexNode rexNode)