org.apache.hadoop.hive.ql.exec.vector.ptf

Class VectorPTFGroupBatches

java.lang.Object

org.apache.hadoop.hive.ql.exec.PTFPartition

org.apache.hadoop.hive.ql.exec.vector.ptf.VectorPTFGroupBatches

public class VectorPTFGroupBatches
extends PTFPartition

This class is encapsulates one or more VectorizedRowBatch of a PTF group.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.exec.PTFPartition

PTFPartition.PTFPartitionIterator<T>

Field Summary

Fields inherited from class org.apache.hadoop.hive.ql.exec.PTFPartition

boundaryCache, valueCache

Constructor Summary

Constructors

Constructor and Description
VectorPTFGroupBatches(org.apache.hadoop.conf.Configuration hconf, int vectorizedPTFMaxMemoryBufferingBatchCount)

Method Summary

Modifier and Type	Method and Description
void	bufferGroupBatch(VectorizedRowBatch batch, boolean isLastGroupBatch) Buffers a batch at the end of bufferedBatches, and if it's full, spills a block of batches to the disk.
void	cleanupPartition() This should be called, when all of the batches were processed for a group (all of the evaluators have been evaluated for this group).
void	evaluateStreamingGroupBatch(VectorizedRowBatch batch, boolean isLastGroupBatch)
void	fillGroupResultsAndForward(VectorPTFOperator vecPTFOperator, Object[] partitionKey)
void	finishPartition()
Object[]	getAt(int i) Finds and element by a partition-level row index.
org.apache.hadoop.hive.ql.exec.vector.ptf.VectorPTFGroupBatches.RowPositionInBatch	getPosition(int i)
Object	getValue(int row, int col) Returns the value at row/col.
Object	getValueAndEvaluateInputExpression(VectorPTFEvaluatorBase evaluator, int row, int col)
void	init(VectorPTFEvaluatorBase[] evaluators, int[] outputProjectionColumnMap, int[] bufferedColumnMap, TypeInfo[] bufferedTypeInfos, int[] orderColumnMap, int[] keyWithoutOrderColumnMap, VectorizedRowBatch overflowBatch)
void	jumpToLastBlock()
void	resetEvaluators()
Object	runEvaluatorOnRange(VectorPTFEvaluatorBase evaluator, Range range)
Object	runEvaluatorOnRange(VectorPTFEvaluatorBase evaluator, Range range, boolean resetAfterCalculation)
int	size() Overrides PTFPartition.size(), this is supposed to return the amount of records in the partition.

Methods inherited from class org.apache.hadoop.hive.ql.exec.PTFPartition

append, close, create, createRolling, getBoundaryCache, getInputOI, getOutputOI, getSerDe, initBoundaryCache, initValueCache, iterator, range, range, reset, setupPartitionOutputOI

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

VectorPTFGroupBatches

public VectorPTFGroupBatches(org.apache.hadoop.conf.Configuration hconf,
                             int vectorizedPTFMaxMemoryBufferingBatchCount)

Method Detail

init

public void init(VectorPTFEvaluatorBase[] evaluators,
                 int[] outputProjectionColumnMap,
                 int[] bufferedColumnMap,
                 TypeInfo[] bufferedTypeInfos,
                 int[] orderColumnMap,
                 int[] keyWithoutOrderColumnMap,
                 VectorizedRowBatch overflowBatch)

getAt

public Object[] getAt(int i)
               throws HiveException

Finds and element by a partition-level row index. This search can use per-batch indices. Overrides PTFPartition.getAt() but here in the vectorized codepath (as opposed to non-vectorized case), getAt only returns the ordering columns. This is because most of the time getAt is used heavily for range scanning where only the ordering columns are needed. As VectorizedRowBatch maintains columns, extracting a full row needs more computational effort than just 1-2 ordering columns. Accessing a random element is possible via getValue(row, col).

Overrides:

getAt in class PTFPartition

Throws:

HiveException

getValue

public Object getValue(int row,
                       int col)
                throws HiveException

Returns the value at row/col. Row index is absolute inside a PTF partition, col should point to the remapped column index according to bufferedColumnMap. The input expressions of the evaluators are already aware of this remapped column as they are changed in VectorPTFOperator.initBufferedColumns.

Parameters:

row -

col -

Returns:

Throws:

HiveException

getValueAndEvaluateInputExpression

public Object getValueAndEvaluateInputExpression(VectorPTFEvaluatorBase evaluator,
                                                 int row,
                                                 int col)
                                          throws HiveException

Throws:

HiveException

getPosition

public org.apache.hadoop.hive.ql.exec.vector.ptf.VectorPTFGroupBatches.RowPositionInBatch getPosition(int i)
                                                                                               throws HiveException

Throws:

HiveException

size

public int size()

Overrides PTFPartition.size(), this is supposed to return the amount of records in the partition.

Overrides:

size in class PTFPartition

finishPartition

public void finishPartition()
                     throws HiveException

Throws:

HiveException

cleanupPartition

public void cleanupPartition()

This should be called, when all of the batches were processed for a group (all of the evaluators have been evaluated for this group). The data is not needed anymore.

bufferGroupBatch

public void bufferGroupBatch(VectorizedRowBatch batch,
                             boolean isLastGroupBatch)
                      throws HiveException

Buffers a batch at the end of bufferedBatches, and if it's full, spills a block of batches to the disk.

Parameters:

batch -

Throws:

HiveException

resetEvaluators

public void resetEvaluators()

fillGroupResultsAndForward

public void fillGroupResultsAndForward(VectorPTFOperator vecPTFOperator,
                                       Object[] partitionKey)
                                throws HiveException

Throws:

HiveException

evaluateStreamingGroupBatch

public void evaluateStreamingGroupBatch(VectorizedRowBatch batch,
                                        boolean isLastGroupBatch)
                                 throws HiveException

Throws:

HiveException

runEvaluatorOnRange

public Object runEvaluatorOnRange(VectorPTFEvaluatorBase evaluator,
                                  Range range)
                           throws HiveException

Throws:

HiveException

runEvaluatorOnRange

public Object runEvaluatorOnRange(VectorPTFEvaluatorBase evaluator,
                                  Range range,
                                  boolean resetAfterCalculation)
                           throws HiveException

Throws:

HiveException

jumpToLastBlock

public void jumpToLastBlock()
                     throws HiveException

Throws:

HiveException