org.apache.hadoop.hive.ql.optimizer

Class SortedDynPartitionOptimizer

java.lang.Object

org.apache.hadoop.hive.ql.optimizer.Transform

org.apache.hadoop.hive.ql.optimizer.SortedDynPartitionOptimizer

public class SortedDynPartitionOptimizer
extends Transform

When dynamic partitioning (with or without bucketing and sorting) is enabled, this optimization sorts the records on partition, bucket and sort columns respectively before inserting records into the destination table. This enables reducers to keep only one record writer all the time thereby reducing the the memory pressure on the reducers. Sorting is based on the Dynamic Partitioning context that is already created in the file sink operator. If that contains instructions for custom expression sorting, then this optimizer will disregard any partitioning or bucketing information of the Hive (table format) table, and will arrange the plan solely as per the custom exprs.

Constructor Summary

Constructors

Constructor and Description
SortedDynPartitionOptimizer()

Method Summary

Modifier and Type	Method and Description
ParseContext	transform(ParseContext pCtx) All transformation steps implement this interface.

Methods inherited from class org.apache.hadoop.hive.ql.optimizer.Transform

beginPerfLogging, endPerfLogging, endPerfLogging

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SortedDynPartitionOptimizer

public SortedDynPartitionOptimizer()

Method Detail

transform

public ParseContext transform(ParseContext pCtx)
                       throws SemanticException

Description copied from class: Transform

All transformation steps implement this interface.

Specified by:

transform in class Transform

Parameters:

pCtx - input parse context

Returns:

ParseContext

Throws:

SemanticException