org.apache.hadoop.hive.ql.optimizer

Class SharedWorkOptimizer

java.lang.Object

org.apache.hadoop.hive.ql.optimizer.Transform

org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer

public class SharedWorkOptimizer
extends Transform

Shared computation optimizer.

Originally, this rule would find scan operators over the same table in the query plan and merge them if they met some preconditions.

  TS   TS             TS
  |    |     ->   /  \
  Op   Op           Op  Op
 

Now the rule has been extended to find opportunities to other operators downstream, not only a single table scan. TS1 TS2 TS1 TS2 TS1 TS2 | | | | | | | RS | RS | RS \ / \ / -> \ / MapJoin MapJoin MapJoin | | / \ Op Op Op Op

If the extended version of the optimizer is enabled, it can go beyond a work boundary to find reutilization opportunities.

The optimization only works with the Tez execution engine.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SharedWorkOptimizer.Mode SharedWorkOptimization strategy modes

Constructor Summary

Constructors

Constructor and Description
SharedWorkOptimizer()

Method Summary

Modifier and Type	Method and Description
protected boolean	areMergeable(ParseContext pctx, TableScanOperator tsOp1, TableScanOperator tsOp2)
protected void	mergeSchema(TableScanOperator discardableTsOp, TableScanOperator retainableTsOp)
boolean	sharedWorkOptimization(ParseContext pctx, org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.SharedWorkOptimizerCache optimizerCache, List<TableScanOperator> tableScans, SharedWorkOptimizer.Mode mode, boolean schemaMerge) Class wrapping shared work optimizer.
ParseContext	transform(ParseContext pctx) All transformation steps implement this interface.

Methods inherited from class org.apache.hadoop.hive.ql.optimizer.Transform

beginPerfLogging, endPerfLogging, endPerfLogging

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SharedWorkOptimizer

public SharedWorkOptimizer()

Method Detail

transform

public ParseContext transform(ParseContext pctx)
                       throws SemanticException

Description copied from class: Transform

All transformation steps implement this interface.

Specified by:

transform in class Transform

Parameters:

pctx - input parse context

Returns:

ParseContext

Throws:

SemanticException

sharedWorkOptimization

public boolean sharedWorkOptimization(ParseContext pctx,
                                      org.apache.hadoop.hive.ql.optimizer.SharedWorkOptimizer.SharedWorkOptimizerCache optimizerCache,
                                      List<TableScanOperator> tableScans,
                                      SharedWorkOptimizer.Mode mode,
                                      boolean schemaMerge)
                               throws SemanticException

Class wrapping shared work optimizer. This implementation enables merging of TS with different schemas by taking the union of the TableScanDesc.getNeededColumns() and TableScanDesc.getNeededColumnIDs() from both TableScanOperators.

Throws:

SemanticException

areMergeable

protected boolean areMergeable(ParseContext pctx,
                               TableScanOperator tsOp1,
                               TableScanOperator tsOp2)
                        throws SemanticException

Throws:

SemanticException

mergeSchema

protected void mergeSchema(TableScanOperator discardableTsOp,
                           TableScanOperator retainableTsOp)