org.apache.hadoop.hive.ql.exec.vector.mapjoin

Class VectorMapJoinInnerGenerateResultOperator

java.lang.Object

org.apache.hadoop.hive.ql.exec.Operator<T>

org.apache.hadoop.hive.ql.exec.CommonJoinOperator<T>

org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator<MapJoinDesc>

org.apache.hadoop.hive.ql.exec.MapJoinOperator

org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator

org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator

org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinInnerGenerateResultOperator

All Implemented Interfaces:

Serializable, Cloneable, VectorizationContextRegion, VectorizationOperator, Node

Direct Known Subclasses:

VectorMapJoinInnerLongOperator, VectorMapJoinInnerMultiKeyOperator, VectorMapJoinInnerStringOperator

public abstract class VectorMapJoinInnerGenerateResultOperator
extends VectorMapJoinGenerateResultOperator

This class has methods for generating vectorized join results for inner joins. Inner joins use a hash map to lookup the 1 or more small table values. One vector inner join optimization is projecting inner keys. When a key appears in the small table results area, instead of copying or referencing key we just include that key again in the output projection. Another optimization is when an inner join does not have any small table columns in the join result, we use a different variation call inner big only. That variation uses a hash multi-set instead of hash map since there are no values (just a count).

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.exec.Operator

Operator.Counter, Operator.OperatorFunc, Operator.State

Field Summary

Fields

Modifier and Type	Field and Description
protected int[]	allMatches
protected int[]	equalKeySeriesAllMatchIndices
protected int[]	equalKeySeriesDuplicateCounts
protected int[]	equalKeySeriesHashMapResultIndices
protected boolean[]	equalKeySeriesIsSingleValue
protected VectorMapJoinHashMapResult[]	hashMapResults
protected int[]	spillHashMapResultIndices
protected int[]	spills

Fields inherited from class org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator

allSmallTableKeyColumnIncluded, allSmallTableKeyColumnNums, batchCounter, bigTableByteColumnVectorColumns, bigTableFilterExpressions, bigTableKeyColumnMap, bigTableKeyColumnNames, bigTableKeyExpressions, bigTableKeyTypeInfos, bigTableRetainColumnMap, bigTableRetainedVectorCopy, bigTableRetainTypeInfos, bigTableValueColumnMap, bigTableValueColumnNames, bigTableValueExpressions, bigTableValueTypeInfos, fullOuterSmallTableKeyMapping, hashTableKeyType, hashTableKind, isOuterJoin, loggingPrefix, needCommonSetup, needFirstBatchSetup, needHashTableSetup, nonOuterSmallTableKeyByteColumnVectorColumns, nonOuterSmallTableKeyColumnMap, nonOuterSmallTableKeyTypeInfos, nonOuterSmallTableKeyVectorCopy, outerSmallTableKeyByteColumnVectorColumns, outerSmallTableKeyColumnMap, outerSmallTableKeyMapping, outerSmallTableKeyVectorCopy, outputProjection, outputTypeInfos, overflowBatch, overflowRepeatedThreshold, posSingleVectorMapJoinSmallTable, projectionMapping, rowCounter, smallTableByteColumnVectorColumns, smallTableKeyOuterVectorDeserializeRow, smallTableValueColumnMap, smallTableValueMapping, smallTableValueVectorDeserializeRow, spillReplayBatch, useOverflowRepeatedThreshold, vContext, vectorDesc, vectorMapJoinHashTable, vectorMapJoinInfo, vectorMapJoinVariation, vOutContext

Fields inherited from class org.apache.hadoop.hive.ql.exec.MapJoinOperator

firstSmallTable, fullOuterBigTableRetainSize, hashMapRowGetters, isFullOuterMapJoin, isTestingNoHashTableLoad, loader, mapJoinTables, matchTracker, spilledMapJoinTables

Fields inherited from class org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator

emptyList, joinKeys, joinKeysObjectInspectors, posBigTable

Fields inherited from class org.apache.hadoop.hive.ql.exec.CommonJoinOperator

alias, aliasFilterTags, aliasFilterTagsNext, condn, countAfterReport, dummyObj, dummyObjVectors, filterMaps, filterTags, forwardCache, heartbeatInterval, intermediate, joinFilterObjectInspectors, joinFilters, joinValues, joinValuesObjectInspectors, joinValuesStandardObjectInspectors, needsPostEvaluation, noOuterJoin, NOTSKIPBIGTABLE, nullsafes, numAliases, offsets, order, residualJoinFilters, residualJoinFiltersOIs, rowContainerPostFilteredOuterJoin, rowContainerStandardObjectInspectors, skipVectors, spillTableDesc, totalSz

Fields inherited from class org.apache.hadoop.hive.ql.exec.Operator

abortOp, asyncInitOperations, cContext, childOperators, childOperatorsArray, childOperatorsTag, conf, CONTEXT_NAME_KEY, done, groupKeyObject, HIVE_COUNTER_CREATED_DYNAMIC_PARTITIONS, HIVE_COUNTER_CREATED_FILES, HIVE_COUNTER_FATAL, id, inputObjInspectors, numRows, operatorId, out, outputObjInspector, parentOperators, reporter, runTimeNumRows, state, statsMap

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	VectorMapJoinInnerGenerateResultOperator() Kryo ctor.
	VectorMapJoinInnerGenerateResultOperator(CompilationOpContext ctx)
	VectorMapJoinInnerGenerateResultOperator(CompilationOpContext ctx, OperatorDesc conf, VectorizationContext vContext, VectorDesc vectorDesc)

Method Summary

Modifier and Type	Method and Description
protected void	commonSetup()
protected void	finishInner(VectorizedRowBatch batch, int allMatchCount, int equalKeySeriesCount, int spillCount, int hashMapResultCount) Generate the inner join output results for one vectorized row batch.
protected void	finishInnerRepeated(VectorizedRowBatch batch, JoinUtil.JoinResult joinResult, VectorMapJoinHashTableResult hashMapResult)
protected void	innerPerBatchSetup(VectorizedRowBatch batch) Do the per-batch setup for an inner join.

Methods inherited from class org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator

closeOp, doSmallTableValueDeserializeRow, forwardBigTableBatch, forwardOverflow, generateHashMapResultMultiValue, generateHashMapResultRepeatedAll, generateHashMapResultSingleValue, initializeOp, intArrayToRangesString, longArrayToRangesString, performValueExpressions, reloadHashTable, reProcessBigTable, spillBatchRepeated, spillHashMapBatch, spillRow, verifyMonotonicallyIncreasing

Methods inherited from class org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator

completeInitializationOp, determineCommonInfo, displayBatchColumns, firstBatchSetup, getHashTableLoader, getInputVectorizationContext, getLoggingPrefix, getLoggingPrefix, getOutputVectorizationContext, getType, getVectorDesc, hashTableSetup, initLoggingPrefix, process, processBatch, setTestMapJoinTableContainer, setupOverflowBatch, setupVOutContext

Methods inherited from class org.apache.hadoop.hive.ql.exec.MapJoinOperator

canSkipJoinProcessing, canSkipReload, cleanUpInputFileChangedOp, endGroup, generateFullOuterSmallTableNoMatches, generateMapMetaData, getCacheKey, getName, getOperatorName, getRefKey, getValueObjectInspectors, isInputFileChangeSensitive, loadHashTable, setMapJoinKey, setMapJoinKeyNoNulls, spillBigTableRow, startGroup

Methods inherited from class org.apache.hadoop.hive.ql.exec.CommonJoinOperator

addToAliasFilterTags, checkAndGenObject, getFilteredValue, getFilterTag, getNextSize, getPosToAliasMap, hasFilter, internalForward, opAllowedAfterMapJoin, opAllowedBeforeMapJoin, reportProgress, setPosToAliasMap

Methods inherited from class org.apache.hadoop.hive.ql.exec.Operator

abort, acceptLimitPushdown, allInitializedParentsAreClosed, areAllParentsInitialized, augmentPlan, batchNeedsClone, cleanUpInputFileChanged, clone, cloneOp, cloneRecursiveChildren, close, columnNamesRowResolvedCanBeObtained, createDummy, defaultEndGroup, defaultStartGroup, dump, dump, flush, flushRecursive, forward, getAdditionalCounters, getChildOperators, getChildren, getColumnExprMap, getCompilationOpContext, getConf, getConfiguration, getCounterName, getDone, getExecContext, getGroupKeyObject, getIdentifier, getInputObjInspectors, getIsReduceSink, getMarker, getNextCntr, getNumChild, getNumParent, getOperatorId, getOpTraits, getOutputObjInspector, getParentOperators, getReduceOutputName, getSchema, getStatistics, getStats, initEvaluators, initEvaluators, initEvaluatorsAndReturnStruct, initialize, initialize, initializeChildren, initializeLocalWork, initOperatorId, isUseBucketizedHiveInputFormat, jobClose, jobCloseOp, logicalEquals, logicalEqualsTree, logStats, opAllowedBeforeSortMergeJoin, opAllowedConvertMapJoin, passExecContext, preorderMap, processGroup, removeChild, removeChildAndAdoptItsChildren, removeParent, removeParents, replaceChild, replaceParent, replaceTabAlias, reset, setAlias, setChildOperators, setColumnExprMap, setCompilationOpContext, setConf, setDone, setExecContext, setGroupKeyObject, setInputContext, setInputObjInspectors, setMarker, setNextVectorBatchGroupStatus, setOpTraits, setOutputCollector, setParentOperators, setReporter, setSchema, setStatistics, setUseBucketizedHiveInputFormat, supportAutomaticSortMergeJoin, supportSkewJoinOptimization, supportUnionRemoveOptimization, toString, toString, toString, vectorForward

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

hashMapResults

protected transient VectorMapJoinHashMapResult[] hashMapResults

allMatches

protected transient int[] allMatches

equalKeySeriesHashMapResultIndices

protected transient int[] equalKeySeriesHashMapResultIndices

equalKeySeriesAllMatchIndices

protected transient int[] equalKeySeriesAllMatchIndices

equalKeySeriesIsSingleValue

protected transient boolean[] equalKeySeriesIsSingleValue

equalKeySeriesDuplicateCounts

protected transient int[] equalKeySeriesDuplicateCounts

spills

protected transient int[] spills

spillHashMapResultIndices

protected transient int[] spillHashMapResultIndices

Constructor Detail

VectorMapJoinInnerGenerateResultOperator

protected VectorMapJoinInnerGenerateResultOperator()

Kryo ctor.

VectorMapJoinInnerGenerateResultOperator

public VectorMapJoinInnerGenerateResultOperator(CompilationOpContext ctx)

VectorMapJoinInnerGenerateResultOperator

public VectorMapJoinInnerGenerateResultOperator(CompilationOpContext ctx,
                                                OperatorDesc conf,
                                                VectorizationContext vContext,
                                                VectorDesc vectorDesc)
                                         throws HiveException

Throws:

HiveException

Method Detail

commonSetup

protected void commonSetup()
                    throws HiveException

Overrides:

commonSetup in class VectorMapJoinCommonOperator

Throws:

HiveException

innerPerBatchSetup

protected void innerPerBatchSetup(VectorizedRowBatch batch)

Do the per-batch setup for an inner join.

finishInner

protected void finishInner(VectorizedRowBatch batch,
                           int allMatchCount,
                           int equalKeySeriesCount,
                           int spillCount,
                           int hashMapResultCount)
                    throws HiveException,
                           IOException

Generate the inner join output results for one vectorized row batch.

Parameters:

batch - The big table batch with any matching and any non matching rows both as selected in use.

allMatchCount - Number of matches in allMatches.

equalKeySeriesCount - Number of single value matches.

spillCount - Number of spills in spills.

hashMapResultCount - Number of entries in hashMapResults.

Throws:

HiveException

IOException

finishInnerRepeated

protected void finishInnerRepeated(VectorizedRowBatch batch,
                                   JoinUtil.JoinResult joinResult,
                                   VectorMapJoinHashTableResult hashMapResult)
                            throws HiveException,
                                   IOException

Throws:

HiveException

IOException