org.apache.hadoop.hive.ql.optimizer.calcite.rules.views

Class HiveAggregateInsertDeleteIncrementalRewritingRule

java.lang.Object

org.apache.calcite.plan.RelOptRule

org.apache.hadoop.hive.ql.optimizer.calcite.rules.views.HiveAggregateIncrementalRewritingRuleBase<org.apache.hadoop.hive.ql.optimizer.calcite.rules.views.HiveAggregateInsertDeleteIncrementalRewritingRule.IncrementalComputePlanWithDeletedRows>

org.apache.hadoop.hive.ql.optimizer.calcite.rules.views.HiveAggregateInsertDeleteIncrementalRewritingRule

public class HiveAggregateInsertDeleteIncrementalRewritingRule
extends HiveAggregateIncrementalRewritingRuleBase<org.apache.hadoop.hive.ql.optimizer.calcite.rules.views.HiveAggregateInsertDeleteIncrementalRewritingRule.IncrementalComputePlanWithDeletedRows>

This rule will perform a rewriting to prepare the plan for incremental view maintenance in case there exist aggregation operator, so we can avoid the INSERT OVERWRITE and use a MERGE statement instead.

 In particular, the INSERT OVERWRITE maintenance will look like this
 (in SQL):
 INSERT OVERWRITE mv
 SELECT a, b, SUM(s) as s, SUM(c) AS c
 FROM (
   SELECT * from mv --OLD DATA
   UNION ALL
   SELECT a, b, SUM(x) AS s, COUNT(*) AS c --NEW DATA
   FROM TAB_A
   JOIN TAB_B ON (TAB_A.a = TAB_B.z)
   WHERE TAB_A.ROW_ID > 5
   GROUP BY a, b) inner_subq
 GROUP BY a, b;
 

We need to transform that into:

 MERGE INTO mv
 USING (
   SELECT a, b, SUM(x) AS s, COUNT(*) AS c --NEW DATA
   FROM TAB_A
   JOIN TAB_B ON (TAB_A.a = TAB_B.z)
   WHERE TAB_A.ROW_ID > 5
   GROUP BY a, b) source
 ON (mv.a <=> source.a AND mv.b <=> source.b)
 WHEN MATCHED AND mv.c + source.c <> 0
   THEN UPDATE SET mv.s = mv.s + source.s, mv.c = mv.c + source.c
 WHEN MATCHED AND countStar = 0 THEN DELETE
 WHEN NOT MATCHED
   THEN INSERT VALUES (source.a, source.b, s, c);
 

To be precise, we need to convert it into a MERGE rewritten as:

 FROM (select *, true flag from mv) mv right outer join _source_ source
 ON (mv.a <=> source.a AND mv.b <=> source.b)
 INSERT INTO TABLE mv                                       <- (insert new rows into the view)
   SELECT source.a, source.b, s, c
   WHERE mv.flag IS NULL
 INSERT INTO TABLE mv                                       <- (update existing rows in the view)
   SELECT mv.ROW__ID, source.a, source.b,
     CASE WHEN mv.s IS NULL AND source.s IS NULL THEN NULL  <- (use case expression to handle nulls from both sides)
          WHEN mv.s IS NULL THEN source.s
          WHEN source.s IS NULL THEN mv.s
          ELSE mv.s + source.s END,
     CASE WHEN mv.s IS NULL AND source.s IS NULL THEN NULL
          WHEN mv.s IS NULL THEN source.s
          WHEN source.s IS NULL THEN mv.s
          ELSE mv.s + source.s END countStar,
   WHERE mv.flag AND countStar <> 0
   SORT BY mv.ROW__ID;
 INSERT INTO TABLE mv                                       <- (delete from the views)
   SELECT mv.ROW__ID
   WHERE mv.flag AND countStar = 0
   SORT BY mv.ROW__ID;
 

See Also:

AlterMaterializedViewRebuildAnalyzer

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.optimizer.calcite.rules.views.HiveAggregateIncrementalRewritingRuleBase

HiveAggregateIncrementalRewritingRuleBase.IncrementalComputePlan

Nested classes/interfaces inherited from class org.apache.calcite.plan.RelOptRule

org.apache.calcite.plan.RelOptRule.ConverterRelOptRuleOperand

Field Summary

Fields

Modifier and Type	Field and Description
static HiveAggregateInsertDeleteIncrementalRewritingRule	INSTANCE

Fields inherited from class org.apache.calcite.plan.RelOptRule

description, operands, relBuilderFactory

Method Summary

Modifier and Type	Method and Description
protected org.apache.calcite.rex.RexNode	createFilterCondition(org.apache.hadoop.hive.ql.optimizer.calcite.rules.views.HiveAggregateInsertDeleteIncrementalRewritingRule.IncrementalComputePlanWithDeletedRows incrementalComputePlan, org.apache.calcite.rex.RexNode flagNode, List<org.apache.calcite.rex.RexNode> projExprs, org.apache.calcite.tools.RelBuilder relBuilder)
protected org.apache.hadoop.hive.ql.optimizer.calcite.rules.views.HiveAggregateInsertDeleteIncrementalRewritingRule.IncrementalComputePlanWithDeletedRows	createJoinRightInput(org.apache.calcite.plan.RelOptRuleCall call)

Methods inherited from class org.apache.hadoop.hive.ql.optimizer.calcite.rules.views.HiveAggregateIncrementalRewritingRuleBase

createAggregateNode, onMatch

Methods inherited from class org.apache.calcite.plan.RelOptRule

any, convert, convert, convertList, convertOperand, convertOperand, equals, equals, getOperand, getOperands, getOutConvention, getOutTrait, hashCode, matches, none, operand, operand, operand, operand, operand, operandJ, operandJ, some, toString, unordered

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

INSTANCE

public static final HiveAggregateInsertDeleteIncrementalRewritingRule INSTANCE

Method Detail

createJoinRightInput

protected org.apache.hadoop.hive.ql.optimizer.calcite.rules.views.HiveAggregateInsertDeleteIncrementalRewritingRule.IncrementalComputePlanWithDeletedRows createJoinRightInput(org.apache.calcite.plan.RelOptRuleCall call)

Specified by:

createJoinRightInput in class HiveAggregateIncrementalRewritingRuleBase<org.apache.hadoop.hive.ql.optimizer.calcite.rules.views.HiveAggregateInsertDeleteIncrementalRewritingRule.IncrementalComputePlanWithDeletedRows>

createFilterCondition

protected org.apache.calcite.rex.RexNode createFilterCondition(org.apache.hadoop.hive.ql.optimizer.calcite.rules.views.HiveAggregateInsertDeleteIncrementalRewritingRule.IncrementalComputePlanWithDeletedRows incrementalComputePlan,
                                                               org.apache.calcite.rex.RexNode flagNode,
                                                               List<org.apache.calcite.rex.RexNode> projExprs,
                                                               org.apache.calcite.tools.RelBuilder relBuilder)

Specified by:

createFilterCondition in class HiveAggregateIncrementalRewritingRuleBase<org.apache.hadoop.hive.ql.optimizer.calcite.rules.views.HiveAggregateInsertDeleteIncrementalRewritingRule.IncrementalComputePlanWithDeletedRows>