org.apache.hadoop.hive.ql.exec.vector.mapjoin

Class VectorMapJoinLeftSemiGenerateResultOperator

java.lang.Object

org.apache.hadoop.hive.ql.exec.Operator<T>

org.apache.hadoop.hive.ql.exec.CommonJoinOperator<T>

org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator<MapJoinDesc>

org.apache.hadoop.hive.ql.exec.MapJoinOperator

org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator

org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator

org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinLeftSemiGenerateResultOperator

All Implemented Interfaces:

Serializable, Cloneable, VectorizationContextRegion, VectorizationOperator, Node

Direct Known Subclasses:

VectorMapJoinLeftSemiLongOperator, VectorMapJoinLeftSemiMultiKeyOperator, VectorMapJoinLeftSemiStringOperator

public abstract class VectorMapJoinLeftSemiGenerateResultOperator
extends VectorMapJoinGenerateResultOperator

This class has methods for generating vectorized join results for left semi joins. The big difference between inner joins and left semi joins is existence testing. Inner joins use a hash map to lookup the 1 or more small table values. Left semi joins are a specialized join for outputting big table rows whose key exists in the small table. No small table values are needed for left semi join since they would be empty. So, we use a hash set as the hash table. Hash sets just report whether a key exists. This is a big performance optimization.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.exec.Operator

Operator.Counter, Operator.OperatorFunc, Operator.State

Field Summary

Fields

Modifier and Type	Field and Description
protected int[]	allMatchs
protected VectorMapJoinHashSetResult[]	hashSetResults
protected int[]	spillHashMapResultIndices
protected int[]	spills

Fields inherited from class org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator

allSmallTableKeyColumnIncluded, allSmallTableKeyColumnNums, batchCounter, bigTableByteColumnVectorColumns, bigTableFilterExpressions, bigTableKeyColumnMap, bigTableKeyColumnNames, bigTableKeyExpressions, bigTableKeyTypeInfos, bigTableRetainColumnMap, bigTableRetainedVectorCopy, bigTableRetainTypeInfos, bigTableValueColumnMap, bigTableValueColumnNames, bigTableValueExpressions, bigTableValueTypeInfos, fullOuterSmallTableKeyMapping, hashTableKeyType, hashTableKind, isOuterJoin, loggingPrefix, needCommonSetup, needFirstBatchSetup, needHashTableSetup, nonOuterSmallTableKeyByteColumnVectorColumns, nonOuterSmallTableKeyColumnMap, nonOuterSmallTableKeyTypeInfos, nonOuterSmallTableKeyVectorCopy, outerSmallTableKeyByteColumnVectorColumns, outerSmallTableKeyColumnMap, outerSmallTableKeyMapping, outerSmallTableKeyVectorCopy, outputProjection, outputTypeInfos, overflowBatch, overflowRepeatedThreshold, posSingleVectorMapJoinSmallTable, projectionMapping, rowCounter, smallTableByteColumnVectorColumns, smallTableKeyOuterVectorDeserializeRow, smallTableValueColumnMap, smallTableValueMapping, smallTableValueVectorDeserializeRow, spillReplayBatch, useOverflowRepeatedThreshold, vContext, vectorDesc, vectorMapJoinHashTable, vectorMapJoinInfo, vectorMapJoinVariation, vOutContext

Fields inherited from class org.apache.hadoop.hive.ql.exec.MapJoinOperator

firstSmallTable, fullOuterBigTableRetainSize, hashMapRowGetters, isFullOuterMapJoin, isTestingNoHashTableLoad, loader, mapJoinTables, matchTracker, spilledMapJoinTables

Fields inherited from class org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator

emptyList, joinKeys, joinKeysObjectInspectors, posBigTable

Fields inherited from class org.apache.hadoop.hive.ql.exec.CommonJoinOperator

alias, aliasFilterTags, aliasFilterTagsNext, condn, countAfterReport, dummyObj, dummyObjVectors, filterMaps, filterTags, forwardCache, heartbeatInterval, intermediate, joinFilterObjectInspectors, joinFilters, joinValues, joinValuesObjectInspectors, joinValuesStandardObjectInspectors, needsPostEvaluation, noOuterJoin, NOTSKIPBIGTABLE, nullsafes, numAliases, offsets, order, residualJoinFilters, residualJoinFiltersOIs, rowContainerPostFilteredOuterJoin, rowContainerStandardObjectInspectors, skipVectors, spillTableDesc, totalSz

Fields inherited from class org.apache.hadoop.hive.ql.exec.Operator

abortOp, asyncInitOperations, cContext, childOperators, childOperatorsArray, childOperatorsTag, conf, CONTEXT_NAME_KEY, done, groupKeyObject, HIVE_COUNTER_CREATED_DYNAMIC_PARTITIONS, HIVE_COUNTER_CREATED_FILES, HIVE_COUNTER_FATAL, id, inputObjInspectors, numRows, operatorId, out, outputObjInspector, parentOperators, reporter, runTimeNumRows, state, statsMap

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	VectorMapJoinLeftSemiGenerateResultOperator() Kryo ctor.
	VectorMapJoinLeftSemiGenerateResultOperator(CompilationOpContext ctx)
	VectorMapJoinLeftSemiGenerateResultOperator(CompilationOpContext ctx, OperatorDesc conf, VectorizationContext vContext, VectorDesc vectorDesc)

Method Summary

Modifier and Type	Method and Description
protected void	commonSetup()
protected void	finishLeftSemi(VectorizedRowBatch batch, int allMatchCount, int spillCount, VectorMapJoinHashTableResult[] hashTableResults) Generate the left semi join output results for one vectorized row batch.
protected void	finishLeftSemiRepeated(VectorizedRowBatch batch, JoinUtil.JoinResult joinResult, VectorMapJoinHashTableResult hashSetResult)
protected int	generateHashSetResultRepeatedAll(VectorizedRowBatch batch) Generate the left semi join output results for one vectorized row batch with a repeated key.

Methods inherited from class org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator

closeOp, doSmallTableValueDeserializeRow, forwardBigTableBatch, forwardOverflow, generateHashMapResultMultiValue, generateHashMapResultRepeatedAll, generateHashMapResultSingleValue, initializeOp, intArrayToRangesString, longArrayToRangesString, performValueExpressions, reloadHashTable, reProcessBigTable, spillBatchRepeated, spillHashMapBatch, spillRow, verifyMonotonicallyIncreasing

Methods inherited from class org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator

completeInitializationOp, determineCommonInfo, displayBatchColumns, firstBatchSetup, getHashTableLoader, getInputVectorizationContext, getLoggingPrefix, getLoggingPrefix, getOutputVectorizationContext, getType, getVectorDesc, hashTableSetup, initLoggingPrefix, process, processBatch, setTestMapJoinTableContainer, setupOverflowBatch, setupVOutContext

Methods inherited from class org.apache.hadoop.hive.ql.exec.MapJoinOperator

canSkipJoinProcessing, canSkipReload, cleanUpInputFileChangedOp, endGroup, generateFullOuterSmallTableNoMatches, generateMapMetaData, getCacheKey, getName, getOperatorName, getRefKey, getValueObjectInspectors, isInputFileChangeSensitive, loadHashTable, setMapJoinKey, setMapJoinKeyNoNulls, spillBigTableRow, startGroup

Methods inherited from class org.apache.hadoop.hive.ql.exec.CommonJoinOperator

addToAliasFilterTags, checkAndGenObject, getFilteredValue, getFilterTag, getNextSize, getPosToAliasMap, hasFilter, internalForward, opAllowedAfterMapJoin, opAllowedBeforeMapJoin, reportProgress, setPosToAliasMap

Methods inherited from class org.apache.hadoop.hive.ql.exec.Operator

abort, acceptLimitPushdown, allInitializedParentsAreClosed, areAllParentsInitialized, augmentPlan, cleanUpInputFileChanged, clone, cloneOp, cloneRecursiveChildren, close, columnNamesRowResolvedCanBeObtained, createDummy, defaultEndGroup, defaultStartGroup, dump, dump, flush, flushRecursive, forward, getAdditionalCounters, getChildOperators, getChildren, getColumnExprMap, getCompilationOpContext, getConf, getConfiguration, getCounterName, getDone, getExecContext, getGroupKeyObject, getIdentifier, getInputObjInspectors, getIsReduceSink, getMarker, getNextCntr, getNumChild, getNumParent, getOperatorId, getOpTraits, getOutputObjInspector, getParentOperators, getReduceOutputName, getSchema, getStatistics, getStats, initEvaluators, initEvaluators, initEvaluatorsAndReturnStruct, initialize, initialize, initializeChildren, initializeLocalWork, initOperatorId, isUseBucketizedHiveInputFormat, jobClose, jobCloseOp, logicalEquals, logicalEqualsTree, logStats, opAllowedBeforeSortMergeJoin, opAllowedConvertMapJoin, passExecContext, preorderMap, processGroup, removeChild, removeChildAndAdoptItsChildren, removeParent, removeParents, replaceChild, replaceParent, replaceTabAlias, reset, setAlias, setChildOperators, setColumnExprMap, setCompilationOpContext, setConf, setDone, setExecContext, setGroupKeyObject, setInputContext, setInputObjInspectors, setMarker, setNextVectorBatchGroupStatus, setOpTraits, setOutputCollector, setParentOperators, setReporter, setSchema, setStatistics, setUseBucketizedHiveInputFormat, supportAutomaticSortMergeJoin, supportSkewJoinOptimization, supportUnionRemoveOptimization, toString, toString, toString, vectorForward

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

hashSetResults

protected transient VectorMapJoinHashSetResult[] hashSetResults

allMatchs

protected transient int[] allMatchs

spills

protected transient int[] spills

spillHashMapResultIndices

protected transient int[] spillHashMapResultIndices

Constructor Detail

VectorMapJoinLeftSemiGenerateResultOperator

protected VectorMapJoinLeftSemiGenerateResultOperator()

Kryo ctor.

VectorMapJoinLeftSemiGenerateResultOperator

public VectorMapJoinLeftSemiGenerateResultOperator(CompilationOpContext ctx)

VectorMapJoinLeftSemiGenerateResultOperator

public VectorMapJoinLeftSemiGenerateResultOperator(CompilationOpContext ctx,
                                                   OperatorDesc conf,
                                                   VectorizationContext vContext,
                                                   VectorDesc vectorDesc)
                                            throws HiveException

Throws:

HiveException

Method Detail

commonSetup

protected void commonSetup()
                    throws HiveException

Overrides:

commonSetup in class VectorMapJoinCommonOperator

Throws:

HiveException

finishLeftSemi

protected void finishLeftSemi(VectorizedRowBatch batch,
                              int allMatchCount,
                              int spillCount,
                              VectorMapJoinHashTableResult[] hashTableResults)
                       throws HiveException,
                              IOException

Generate the left semi join output results for one vectorized row batch.

Parameters:

batch - The big table batch with any matching and any non matching rows both as selected in use.

allMatchCount - Number of matches in allMatchs.

spillCount - Number of spills in spills.

hashTableResults - The array of all hash table results for the batch. We need the VectorMapJoinHashTableResult for the spill information.

Throws:

HiveException

IOException

generateHashSetResultRepeatedAll

protected int generateHashSetResultRepeatedAll(VectorizedRowBatch batch)
                                        throws HiveException

Generate the left semi join output results for one vectorized row batch with a repeated key.

Parameters:

batch - The big table batch whose repeated key matches.

Throws:

HiveException

finishLeftSemiRepeated

protected void finishLeftSemiRepeated(VectorizedRowBatch batch,
                                      JoinUtil.JoinResult joinResult,
                                      VectorMapJoinHashTableResult hashSetResult)
                               throws HiveException,
                                      IOException

Throws:

HiveException

IOException