org.apache.hadoop.hive.ql.optimizer.stats.annotation

Class StatsRulesProcFactory.FilterStatsRule

java.lang.Object

org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory.DefaultStatsRule

org.apache.hadoop.hive.ql.optimizer.stats.annotation.StatsRulesProcFactory.FilterStatsRule

All Implemented Interfaces:

NodeProcessor, SemanticNodeProcessor

Direct Known Subclasses:

StatsRulesProcFactory.JoinStatsRule

Enclosing class:

StatsRulesProcFactory

public static class StatsRulesProcFactory.FilterStatsRule
extends StatsRulesProcFactory.DefaultStatsRule
implements SemanticNodeProcessor

FILTER operator does not change the average row size but it does change the number of rows emitted. The reduction in the number of rows emitted is dependent on the filter expression. Notations:

• T(S) - Number of tuples in relations S
• V(S,A) - Number of distinct values of attribute A in relation S

Rules:

• Column equals a constant T(S) = T(R) / V(R,A)
• Inequality conditions T(S) = T(R) / 3
• Not equals comparison - Simple formula T(S) = T(R)
• - Alternate formula T(S) = T(R) (V(R,A) - 1) / V(R,A)
• NOT condition T(S) = 1 - T(S'), where T(S') is the satisfying condition
• Multiple AND conditions Cascadingly apply the rules 1 to 3 (order doesn't matter)
• Multiple OR conditions - Simple formula is to evaluate conditions independently and sum the results T(S) = m1 + m2
• - Alternate formula T(S) = T(R) * ( 1 - ( 1 - m1/T(R) ) * ( 1 - m2/T(R) ))
  where, m1 is the number of tuples that satisfy condition1 and m2 is the number of tuples that satisfy condition2

Worst case: If no column statistics are available, then evaluation of predicate expression will assume worst case (i.e; half the input rows) for each of predicate expression.
For more information, refer 'Estimating The Cost Of Operations' chapter in "Database Systems: The Complete Book" by Garcia-Molina et. al.

Constructor Summary

Constructors

Constructor and Description
FilterStatsRule()

Method Summary

Modifier and Type	Method and Description
protected long	evaluateExpression(Statistics stats, ExprNodeDesc pred, AnnotateStatsProcCtx aspCtx, List<String> neededCols, Operator<?> op, long currNumRows)
Object	process(Node nd, Stack<Node> stack, NodeProcessorCtx procCtx, Object... nodeOutputs) Generic process for all ops that don't have specific implementations.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FilterStatsRule

public FilterStatsRule()

Method Detail

process

public Object process(Node nd,
                      Stack<Node> stack,
                      NodeProcessorCtx procCtx,
                      Object... nodeOutputs)
               throws SemanticException

Description copied from interface: SemanticNodeProcessor

Generic process for all ops that don't have specific implementations.

Specified by:

process in interface NodeProcessor

Specified by:

process in interface SemanticNodeProcessor

Overrides:

process in class StatsRulesProcFactory.DefaultStatsRule

Parameters:

nd - operator to process

procCtx - operator processor context

nodeOutputs - A variable argument list of outputs from other nodes in the walk

Returns:

Object to be returned by the process call

Throws:

SemanticException

evaluateExpression

protected long evaluateExpression(Statistics stats,
                                  ExprNodeDesc pred,
                                  AnnotateStatsProcCtx aspCtx,
                                  List<String> neededCols,
                                  Operator<?> op,
                                  long currNumRows)
                           throws SemanticException

Throws:

SemanticException