org.apache.hadoop.hive.ql.exec.vector.mapjoin

Class VectorMapJoinGenerateResultOperator

java.lang.Object

org.apache.hadoop.hive.ql.exec.Operator<T>

org.apache.hadoop.hive.ql.exec.CommonJoinOperator<T>

org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator<MapJoinDesc>

org.apache.hadoop.hive.ql.exec.MapJoinOperator

org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator

org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator

All Implemented Interfaces:

Serializable, Cloneable, VectorizationContextRegion, VectorizationOperator, Node

Direct Known Subclasses:

VectorMapJoinAntiJoinGenerateResultOperator, VectorMapJoinInnerBigOnlyGenerateResultOperator, VectorMapJoinInnerGenerateResultOperator, VectorMapJoinLeftSemiGenerateResultOperator, VectorMapJoinOuterGenerateResultOperator

public abstract class VectorMapJoinGenerateResultOperator
extends VectorMapJoinCommonOperator

This class has methods for generating vectorized join results and forwarding batchs. In some cases when can forward the big table batch by setting scratch columns with small table results and then making use of our output projection to pick out all the output result columns. This can improve performance by avoiding copying big table values. So, we will use the big table batch's selected in use to represent those rows. At the same time, some output results need to be formed in the overflow batch. For example, to form N x M cross product output results. In this case, we will copy big table values into the overflow batch and set scratch columns in it for small table results. The "schema" of the overflow batch is the same as the big table batch so child operators only need one definition of their input batch. The overflow batch will be typically be forwarded when it gets full, which might not be during a process call. NOTE: Child operators should not remember a received batch.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.exec.Operator

Operator.Counter, Operator.OperatorFunc, Operator.State

Field Summary

Fields inherited from class org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator

allSmallTableKeyColumnIncluded, allSmallTableKeyColumnNums, batchCounter, bigTableByteColumnVectorColumns, bigTableFilterExpressions, bigTableKeyColumnMap, bigTableKeyColumnNames, bigTableKeyExpressions, bigTableKeyTypeInfos, bigTableRetainColumnMap, bigTableRetainedVectorCopy, bigTableRetainTypeInfos, bigTableValueColumnMap, bigTableValueColumnNames, bigTableValueExpressions, bigTableValueTypeInfos, fullOuterSmallTableKeyMapping, hashTableKeyType, hashTableKind, isOuterJoin, loggingPrefix, needCommonSetup, needFirstBatchSetup, needHashTableSetup, nonOuterSmallTableKeyByteColumnVectorColumns, nonOuterSmallTableKeyColumnMap, nonOuterSmallTableKeyTypeInfos, nonOuterSmallTableKeyVectorCopy, outerSmallTableKeyByteColumnVectorColumns, outerSmallTableKeyColumnMap, outerSmallTableKeyMapping, outerSmallTableKeyVectorCopy, outputProjection, outputTypeInfos, overflowBatch, overflowRepeatedThreshold, posSingleVectorMapJoinSmallTable, projectionMapping, rowCounter, smallTableByteColumnVectorColumns, smallTableKeyOuterVectorDeserializeRow, smallTableValueColumnMap, smallTableValueMapping, smallTableValueVectorDeserializeRow, spillReplayBatch, useOverflowRepeatedThreshold, vContext, vectorDesc, vectorMapJoinHashTable, vectorMapJoinInfo, vectorMapJoinVariation, vOutContext

Fields inherited from class org.apache.hadoop.hive.ql.exec.MapJoinOperator

firstSmallTable, fullOuterBigTableRetainSize, hashMapRowGetters, isFullOuterMapJoin, isTestingNoHashTableLoad, loader, mapJoinTables, matchTracker, spilledMapJoinTables

Fields inherited from class org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator

emptyList, joinKeys, joinKeysObjectInspectors, posBigTable

Fields inherited from class org.apache.hadoop.hive.ql.exec.CommonJoinOperator

alias, aliasFilterTags, aliasFilterTagsNext, condn, countAfterReport, dummyObj, dummyObjVectors, filterMaps, filterTags, forwardCache, heartbeatInterval, intermediate, joinFilterObjectInspectors, joinFilters, joinValues, joinValuesObjectInspectors, joinValuesStandardObjectInspectors, needsPostEvaluation, noOuterJoin, NOTSKIPBIGTABLE, nullsafes, numAliases, offsets, order, residualJoinFilters, residualJoinFiltersOIs, rowContainerPostFilteredOuterJoin, rowContainerStandardObjectInspectors, skipVectors, spillTableDesc, totalSz

Fields inherited from class org.apache.hadoop.hive.ql.exec.Operator

abortOp, asyncInitOperations, cContext, childOperators, childOperatorsArray, childOperatorsTag, conf, CONTEXT_NAME_KEY, done, groupKeyObject, HIVE_COUNTER_CREATED_DYNAMIC_PARTITIONS, HIVE_COUNTER_CREATED_FILES, HIVE_COUNTER_FATAL, id, inputObjInspectors, numRows, operatorId, out, outputObjInspector, parentOperators, reporter, runTimeNumRows, state, statsMap

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	VectorMapJoinGenerateResultOperator() Kryo ctor.
	VectorMapJoinGenerateResultOperator(CompilationOpContext ctx)
	VectorMapJoinGenerateResultOperator(CompilationOpContext ctx, OperatorDesc conf, VectorizationContext vContext, VectorDesc vectorDesc)

Method Summary

Modifier and Type	Method and Description
void	closeOp(boolean aborted) On close, make sure a partially filled overflow batch gets forwarded.
protected void	doSmallTableValueDeserializeRow(VectorizedRowBatch batch, int batchIndex, WriteBuffers.ByteSegmentRef byteSegmentRef, VectorMapJoinHashMapResult hashMapResult)
void	forwardBigTableBatch(VectorizedRowBatch batch) Forward the big table batch to the children.
protected void	forwardOverflow() Forward the overflow batch and reset the batch.
protected void	generateHashMapResultMultiValue(VectorizedRowBatch batch, VectorMapJoinHashMapResult hashMapResult, int[] allMatchs, int allMatchesIndex, int duplicateCount) Generate results for a N x M cross product.
protected void	generateHashMapResultRepeatedAll(VectorizedRowBatch batch, VectorMapJoinHashMapResult hashMapResult) Generate optimized results when entire batch key is repeated and it matched the hash map.
protected int	generateHashMapResultSingleValue(VectorizedRowBatch batch, VectorMapJoinHashMapResult hashMapResult, int[] allMatchs, int allMatchesIndex, int duplicateCount, int numSel) Generate join results for a single small table value match.
protected void	initializeOp(org.apache.hadoop.conf.Configuration hconf) Operator specific initialization.
static String	intArrayToRangesString(int[] selection, int size)
static String	longArrayToRangesString(long[] selection, int size)
protected void	performValueExpressions(VectorizedRowBatch batch, int[] allMatchs, int allMatchCount)
protected void	reloadHashTable(byte pos, int partitionId) Reload hashtable from the hash partition.
protected void	reProcessBigTable(int partitionId) Iterate over the big table row container and feed process() with leftover rows
protected void	spillBatchRepeated(VectorizedRowBatch batch, VectorMapJoinHashTableResult hashTableResult)
protected void	spillHashMapBatch(VectorizedRowBatch batch, VectorMapJoinHashTableResult[] hashTableResults, int[] spills, int[] spillHashTableResultIndices, int spillCount)
protected void	spillRow(VectorizedRowBatch batch, int batchIndex, int partitionId)
boolean	verifyMonotonicallyIncreasing(int[] selected, int size)

Methods inherited from class org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator

commonSetup, completeInitializationOp, determineCommonInfo, displayBatchColumns, firstBatchSetup, getHashTableLoader, getInputVectorizationContext, getLoggingPrefix, getLoggingPrefix, getOutputVectorizationContext, getType, getVectorDesc, hashTableSetup, initLoggingPrefix, process, processBatch, setTestMapJoinTableContainer, setupOverflowBatch, setupVOutContext

Methods inherited from class org.apache.hadoop.hive.ql.exec.MapJoinOperator

canSkipJoinProcessing, canSkipReload, cleanUpInputFileChangedOp, endGroup, generateFullOuterSmallTableNoMatches, generateMapMetaData, getCacheKey, getName, getOperatorName, getRefKey, getValueObjectInspectors, isInputFileChangeSensitive, loadHashTable, setMapJoinKey, setMapJoinKeyNoNulls, spillBigTableRow, startGroup

Methods inherited from class org.apache.hadoop.hive.ql.exec.CommonJoinOperator

addToAliasFilterTags, checkAndGenObject, getFilteredValue, getFilterTag, getNextSize, getPosToAliasMap, hasFilter, internalForward, opAllowedAfterMapJoin, opAllowedBeforeMapJoin, reportProgress, setPosToAliasMap

Methods inherited from class org.apache.hadoop.hive.ql.exec.Operator

abort, acceptLimitPushdown, allInitializedParentsAreClosed, areAllParentsInitialized, augmentPlan, cleanUpInputFileChanged, clone, cloneOp, cloneRecursiveChildren, close, columnNamesRowResolvedCanBeObtained, createDummy, defaultEndGroup, defaultStartGroup, dump, dump, flush, flushRecursive, forward, getAdditionalCounters, getChildOperators, getChildren, getColumnExprMap, getCompilationOpContext, getConf, getConfiguration, getCounterName, getDone, getExecContext, getGroupKeyObject, getIdentifier, getInputObjInspectors, getIsReduceSink, getMarker, getNextCntr, getNumChild, getNumParent, getOperatorId, getOpTraits, getOutputObjInspector, getParentOperators, getReduceOutputName, getSchema, getStatistics, getStats, initEvaluators, initEvaluators, initEvaluatorsAndReturnStruct, initialize, initialize, initializeChildren, initializeLocalWork, initOperatorId, isUseBucketizedHiveInputFormat, jobClose, jobCloseOp, logicalEquals, logicalEqualsTree, logStats, opAllowedBeforeSortMergeJoin, opAllowedConvertMapJoin, passExecContext, preorderMap, processGroup, removeChild, removeChildAndAdoptItsChildren, removeParent, removeParents, replaceChild, replaceParent, replaceTabAlias, reset, setAlias, setChildOperators, setColumnExprMap, setCompilationOpContext, setConf, setDone, setExecContext, setGroupKeyObject, setInputContext, setInputObjInspectors, setMarker, setNextVectorBatchGroupStatus, setOpTraits, setOutputCollector, setParentOperators, setReporter, setSchema, setStatistics, setUseBucketizedHiveInputFormat, supportAutomaticSortMergeJoin, supportSkewJoinOptimization, supportUnionRemoveOptimization, toString, toString, toString, vectorForward

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

VectorMapJoinGenerateResultOperator

protected VectorMapJoinGenerateResultOperator()

Kryo ctor.

VectorMapJoinGenerateResultOperator

public VectorMapJoinGenerateResultOperator(CompilationOpContext ctx)

VectorMapJoinGenerateResultOperator

public VectorMapJoinGenerateResultOperator(CompilationOpContext ctx,
                                           OperatorDesc conf,
                                           VectorizationContext vContext,
                                           VectorDesc vectorDesc)
                                    throws HiveException

Throws:

HiveException

Method Detail

initializeOp

protected void initializeOp(org.apache.hadoop.conf.Configuration hconf)
                     throws HiveException

Description copied from class: Operator

Operator specific initialization.

Overrides:

initializeOp in class VectorMapJoinCommonOperator

Throws:

HiveException

performValueExpressions

protected void performValueExpressions(VectorizedRowBatch batch,
                                       int[] allMatchs,
                                       int allMatchCount)
                                throws HiveException

Throws:

HiveException

doSmallTableValueDeserializeRow

protected void doSmallTableValueDeserializeRow(VectorizedRowBatch batch,
                                               int batchIndex,
                                               WriteBuffers.ByteSegmentRef byteSegmentRef,
                                               VectorMapJoinHashMapResult hashMapResult)
                                        throws HiveException

Throws:

HiveException

generateHashMapResultSingleValue

protected int generateHashMapResultSingleValue(VectorizedRowBatch batch,
                                               VectorMapJoinHashMapResult hashMapResult,
                                               int[] allMatchs,
                                               int allMatchesIndex,
                                               int duplicateCount,
                                               int numSel)
                                        throws HiveException,
                                               IOException

Generate join results for a single small table value match.

Parameters:

batch - The big table batch.

hashMapResult - The hash map results for the matching key.

allMatchs - The selection array for all matches key.

allMatchesIndex - Index into allMatches of the matching key we are generating for.

duplicateCount - Number of equal key rows.

numSel - Current number of rows that are remaining in the big table for forwarding.

Returns:

The new count of selected rows.

Throws:

HiveException

IOException

generateHashMapResultMultiValue

protected void generateHashMapResultMultiValue(VectorizedRowBatch batch,
                                               VectorMapJoinHashMapResult hashMapResult,
                                               int[] allMatchs,
                                               int allMatchesIndex,
                                               int duplicateCount)
                                        throws HiveException,
                                               IOException

Generate results for a N x M cross product.

Parameters:

batch - The big table batch.

hashMapResult - The hash map results for the matching key.

allMatchs - The all match selected array that contains (physical) batch indices.

allMatchesIndex - The index of the match key.

duplicateCount - Number of equal key rows.

Throws:

HiveException

IOException

generateHashMapResultRepeatedAll

protected void generateHashMapResultRepeatedAll(VectorizedRowBatch batch,
                                                VectorMapJoinHashMapResult hashMapResult)
                                         throws IOException,
                                                HiveException

Generate optimized results when entire batch key is repeated and it matched the hash map.

Parameters:

batch - The big table batch.

hashMapResult - The hash map results for the repeated key.

Throws:

IOException

HiveException

spillHashMapBatch

protected void spillHashMapBatch(VectorizedRowBatch batch,
                                 VectorMapJoinHashTableResult[] hashTableResults,
                                 int[] spills,
                                 int[] spillHashTableResultIndices,
                                 int spillCount)
                          throws HiveException,
                                 IOException

Throws:

HiveException

IOException

spillRow

protected void spillRow(VectorizedRowBatch batch,
                        int batchIndex,
                        int partitionId)
                 throws HiveException,
                        IOException

Throws:

HiveException

IOException

spillBatchRepeated

protected void spillBatchRepeated(VectorizedRowBatch batch,
                                  VectorMapJoinHashTableResult hashTableResult)
                           throws HiveException,
                                  IOException

Throws:

HiveException

IOException

reloadHashTable

protected void reloadHashTable(byte pos,
                               int partitionId)
                        throws IOException,
                               HiveException,
                               SerDeException,
                               ClassNotFoundException

Description copied from class: MapJoinOperator

Reload hashtable from the hash partition. It can have two steps: 1) Deserialize a serialized hash table, and 2) Merge every key/value pair from small table container into the hash table

Overrides:

reloadHashTable in class MapJoinOperator

Parameters:

pos - position of small table

partitionId - the partition of the small table to be reloaded from

Throws:

IOException

HiveException

SerDeException

ClassNotFoundException

reProcessBigTable

protected void reProcessBigTable(int partitionId)
                          throws HiveException

Description copied from class: MapJoinOperator

Iterate over the big table row container and feed process() with leftover rows

Overrides:

reProcessBigTable in class MapJoinOperator

Parameters:

partitionId - the partition from which to take out spilled big table rows

Throws:

HiveException

forwardBigTableBatch

public void forwardBigTableBatch(VectorizedRowBatch batch)
                          throws HiveException

Forward the big table batch to the children.

Parameters:

batch - The big table batch.

Throws:

HiveException

forwardOverflow

protected void forwardOverflow()
                        throws HiveException

Forward the overflow batch and reset the batch.

Throws:

HiveException

closeOp

public void closeOp(boolean aborted)
             throws HiveException

On close, make sure a partially filled overflow batch gets forwarded.

Overrides:

closeOp in class MapJoinOperator

Throws:

HiveException

verifyMonotonicallyIncreasing

public boolean verifyMonotonicallyIncreasing(int[] selected,
                                             int size)

intArrayToRangesString

public static String intArrayToRangesString(int[] selection,
                                            int size)

longArrayToRangesString

public static String longArrayToRangesString(long[] selection,
                                             int size)