org.apache.hadoop.hive.ql.exec.vector.mapjoin

Class VectorMapJoinOuterGenerateResultOperator

java.lang.Object

org.apache.hadoop.hive.ql.exec.Operator<T>

org.apache.hadoop.hive.ql.exec.CommonJoinOperator<T>

org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator<MapJoinDesc>

org.apache.hadoop.hive.ql.exec.MapJoinOperator

org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator

org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator

org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinOuterGenerateResultOperator

All Implemented Interfaces:

Serializable, Cloneable, VectorizationContextRegion, VectorizationOperator, Node

Direct Known Subclasses:

VectorMapJoinOuterLongOperator, VectorMapJoinOuterMultiKeyOperator, VectorMapJoinOuterStringOperator

public abstract class VectorMapJoinOuterGenerateResultOperator
extends VectorMapJoinGenerateResultOperator

This class has methods for generating vectorized join results for outer joins. The big difference between inner joins and outer joins is the treatment of null and non-matching keys. Inner joins ignore null keys. Outer joins include big table rows with null keys in the result. (Left non-full) outer joins include big table rows that do not match the small table. Small table columns for non-matches will be NULL. Another important difference is filtering. For outer joins to include the necessary rows, filtering must be done after the hash table lookup. That is because filtering does not eliminate rows, but changes them from match to non-matching rows. They will still appear in the join result. One vector outer join optimization is referencing bytes outer keys. When a bytes key appears in the small table results area, instead of copying the bytes key we reference the big table key. Bytes column vectors allow a by reference entry to bytes. It is safe to do a by reference since it is within the same row. Outer join uses a hash map since small table columns can be included in the join result.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.exec.Operator

Operator.Counter, Operator.OperatorFunc, Operator.State

Field Summary

Fields

Modifier and Type	Field and Description
protected int[]	allMatchs
protected int[]	equalKeySeriesAllMatchIndices
protected int[]	equalKeySeriesDuplicateCounts
protected int[]	equalKeySeriesHashMapResultIndices
protected boolean[]	equalKeySeriesIsSingleValue
protected VectorMapJoinHashMapResult[]	hashMapResults
protected int[]	inputSelected
protected int[]	merged
protected int[]	noMatchs
protected int[]	nonSpills
protected int[]	spillHashMapResultIndices
protected int[]	spills

Fields inherited from class org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator

allSmallTableKeyColumnIncluded, allSmallTableKeyColumnNums, batchCounter, bigTableByteColumnVectorColumns, bigTableFilterExpressions, bigTableKeyColumnMap, bigTableKeyColumnNames, bigTableKeyExpressions, bigTableKeyTypeInfos, bigTableRetainColumnMap, bigTableRetainedVectorCopy, bigTableRetainTypeInfos, bigTableValueColumnMap, bigTableValueColumnNames, bigTableValueExpressions, bigTableValueTypeInfos, fullOuterSmallTableKeyMapping, hashTableKeyType, hashTableKind, isOuterJoin, loggingPrefix, needCommonSetup, needFirstBatchSetup, needHashTableSetup, nonOuterSmallTableKeyByteColumnVectorColumns, nonOuterSmallTableKeyColumnMap, nonOuterSmallTableKeyTypeInfos, nonOuterSmallTableKeyVectorCopy, outerSmallTableKeyByteColumnVectorColumns, outerSmallTableKeyColumnMap, outerSmallTableKeyMapping, outerSmallTableKeyVectorCopy, outputProjection, outputTypeInfos, overflowBatch, overflowRepeatedThreshold, posSingleVectorMapJoinSmallTable, projectionMapping, rowCounter, smallTableByteColumnVectorColumns, smallTableKeyOuterVectorDeserializeRow, smallTableValueColumnMap, smallTableValueMapping, smallTableValueVectorDeserializeRow, spillReplayBatch, useOverflowRepeatedThreshold, vContext, vectorDesc, vectorMapJoinHashTable, vectorMapJoinInfo, vectorMapJoinVariation, vOutContext

Fields inherited from class org.apache.hadoop.hive.ql.exec.MapJoinOperator

firstSmallTable, fullOuterBigTableRetainSize, hashMapRowGetters, isFullOuterMapJoin, isTestingNoHashTableLoad, loader, mapJoinTables, matchTracker, spilledMapJoinTables

Fields inherited from class org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator

emptyList, joinKeys, joinKeysObjectInspectors, posBigTable

Fields inherited from class org.apache.hadoop.hive.ql.exec.CommonJoinOperator

alias, aliasFilterTags, aliasFilterTagsNext, condn, countAfterReport, dummyObj, dummyObjVectors, filterMaps, filterTags, forwardCache, heartbeatInterval, intermediate, joinFilterObjectInspectors, joinFilters, joinValues, joinValuesObjectInspectors, joinValuesStandardObjectInspectors, needsPostEvaluation, noOuterJoin, NOTSKIPBIGTABLE, nullsafes, numAliases, offsets, order, residualJoinFilters, residualJoinFiltersOIs, rowContainerPostFilteredOuterJoin, rowContainerStandardObjectInspectors, skipVectors, spillTableDesc, totalSz

Fields inherited from class org.apache.hadoop.hive.ql.exec.Operator

abortOp, asyncInitOperations, cContext, childOperators, childOperatorsArray, childOperatorsTag, conf, CONTEXT_NAME_KEY, done, groupKeyObject, HIVE_COUNTER_CREATED_DYNAMIC_PARTITIONS, HIVE_COUNTER_CREATED_FILES, HIVE_COUNTER_FATAL, id, inputObjInspectors, numRows, operatorId, out, outputObjInspector, parentOperators, reporter, runTimeNumRows, state, statsMap

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	VectorMapJoinOuterGenerateResultOperator() Kryo ctor.
	VectorMapJoinOuterGenerateResultOperator(CompilationOpContext ctx)
	VectorMapJoinOuterGenerateResultOperator(CompilationOpContext ctx, OperatorDesc conf, VectorizationContext vContext, VectorDesc vectorDesc)

Method Summary

Modifier and Type	Method and Description
protected void	commonSetup()
void	finishOuter(VectorizedRowBatch batch, int allMatchCount, int equalKeySeriesCount, boolean atLeastOneNonMatch, boolean inputSelectedInUse, int inputLogicalSize, int spillCount, int hashMapResultCount) Generate the outer join output results for one vectorized row batch.
void	finishOuterRepeated(VectorizedRowBatch batch, JoinUtil.JoinResult joinResult, VectorMapJoinHashMapResult hashMapResult, boolean someRowsFilteredOut, boolean inputSelectedInUse, int inputLogicalSize) Generate the outer join output results for one vectorized row batch with a repeated key.
protected void	fullOuterHashTableSetup()
protected void	fullOuterIntersectHashTableSetup()
protected void	generateFullOuterLongKeySmallTableNoMatches()
protected void	generateFullOuterMultiKeySmallTableNoMatches()
protected void	generateFullOuterSmallTableNoMatches(byte smallTablePos, MapJoinTableContainer substituteSmallTable)
protected void	generateFullOuterStringKeySmallTableNoMatches()
protected void	generateOuterNulls(VectorizedRowBatch batch, int[] noMatchs, int noMatchSize) Generate the non matching outer join output results for one vectorized row batch.
protected void	generateOuterNullsRepeatedAll(VectorizedRowBatch batch) Generate the non-match outer join output results for the whole repeating vectorized row batch.
protected void	outerPerBatchSetup(VectorizedRowBatch batch) Do the per-batch setup for an outer join.

Methods inherited from class org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinGenerateResultOperator

closeOp, doSmallTableValueDeserializeRow, forwardBigTableBatch, forwardOverflow, generateHashMapResultMultiValue, generateHashMapResultRepeatedAll, generateHashMapResultSingleValue, initializeOp, intArrayToRangesString, longArrayToRangesString, performValueExpressions, reloadHashTable, reProcessBigTable, spillBatchRepeated, spillHashMapBatch, spillRow, verifyMonotonicallyIncreasing

Methods inherited from class org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator

completeInitializationOp, determineCommonInfo, displayBatchColumns, firstBatchSetup, getHashTableLoader, getInputVectorizationContext, getLoggingPrefix, getLoggingPrefix, getOutputVectorizationContext, getType, getVectorDesc, hashTableSetup, initLoggingPrefix, process, processBatch, setTestMapJoinTableContainer, setupOverflowBatch, setupVOutContext

Methods inherited from class org.apache.hadoop.hive.ql.exec.MapJoinOperator

canSkipJoinProcessing, canSkipReload, cleanUpInputFileChangedOp, endGroup, generateMapMetaData, getCacheKey, getName, getOperatorName, getRefKey, getValueObjectInspectors, isInputFileChangeSensitive, loadHashTable, setMapJoinKey, setMapJoinKeyNoNulls, spillBigTableRow, startGroup

Methods inherited from class org.apache.hadoop.hive.ql.exec.CommonJoinOperator

addToAliasFilterTags, checkAndGenObject, getFilteredValue, getFilterTag, getNextSize, getPosToAliasMap, hasFilter, internalForward, opAllowedAfterMapJoin, opAllowedBeforeMapJoin, reportProgress, setPosToAliasMap

Methods inherited from class org.apache.hadoop.hive.ql.exec.Operator

abort, acceptLimitPushdown, allInitializedParentsAreClosed, areAllParentsInitialized, augmentPlan, cleanUpInputFileChanged, clone, cloneOp, cloneRecursiveChildren, close, columnNamesRowResolvedCanBeObtained, createDummy, defaultEndGroup, defaultStartGroup, dump, dump, flush, flushRecursive, forward, getAdditionalCounters, getChildOperators, getChildren, getColumnExprMap, getCompilationOpContext, getConf, getConfiguration, getCounterName, getDone, getExecContext, getGroupKeyObject, getIdentifier, getInputObjInspectors, getIsReduceSink, getMarker, getNextCntr, getNumChild, getNumParent, getOperatorId, getOpTraits, getOutputObjInspector, getParentOperators, getReduceOutputName, getSchema, getStatistics, getStats, initEvaluators, initEvaluators, initEvaluatorsAndReturnStruct, initialize, initialize, initializeChildren, initializeLocalWork, initOperatorId, isUseBucketizedHiveInputFormat, jobClose, jobCloseOp, logicalEquals, logicalEqualsTree, logStats, opAllowedBeforeSortMergeJoin, opAllowedConvertMapJoin, passExecContext, preorderMap, processGroup, removeChild, removeChildAndAdoptItsChildren, removeParent, removeParents, replaceChild, replaceParent, replaceTabAlias, reset, setAlias, setChildOperators, setColumnExprMap, setCompilationOpContext, setConf, setDone, setExecContext, setGroupKeyObject, setInputContext, setInputObjInspectors, setMarker, setNextVectorBatchGroupStatus, setOpTraits, setOutputCollector, setParentOperators, setReporter, setSchema, setStatistics, setUseBucketizedHiveInputFormat, supportAutomaticSortMergeJoin, supportSkewJoinOptimization, supportUnionRemoveOptimization, toString, toString, toString, vectorForward

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

hashMapResults

protected transient VectorMapJoinHashMapResult[] hashMapResults

inputSelected

protected transient int[] inputSelected

allMatchs

protected transient int[] allMatchs

equalKeySeriesHashMapResultIndices

protected transient int[] equalKeySeriesHashMapResultIndices

equalKeySeriesAllMatchIndices

protected transient int[] equalKeySeriesAllMatchIndices

equalKeySeriesIsSingleValue

protected transient boolean[] equalKeySeriesIsSingleValue

equalKeySeriesDuplicateCounts

protected transient int[] equalKeySeriesDuplicateCounts

spills

protected transient int[] spills

spillHashMapResultIndices

protected transient int[] spillHashMapResultIndices

nonSpills

protected transient int[] nonSpills

noMatchs

protected transient int[] noMatchs

merged

protected transient int[] merged

Constructor Detail

VectorMapJoinOuterGenerateResultOperator

protected VectorMapJoinOuterGenerateResultOperator()

Kryo ctor.

VectorMapJoinOuterGenerateResultOperator

public VectorMapJoinOuterGenerateResultOperator(CompilationOpContext ctx)

VectorMapJoinOuterGenerateResultOperator

public VectorMapJoinOuterGenerateResultOperator(CompilationOpContext ctx,
                                                OperatorDesc conf,
                                                VectorizationContext vContext,
                                                VectorDesc vectorDesc)
                                         throws HiveException

Throws:

HiveException

Method Detail

commonSetup

protected void commonSetup()
                    throws HiveException

Overrides:

commonSetup in class VectorMapJoinCommonOperator

Throws:

HiveException

outerPerBatchSetup

protected void outerPerBatchSetup(VectorizedRowBatch batch)

Do the per-batch setup for an outer join.

finishOuter

public void finishOuter(VectorizedRowBatch batch,
                        int allMatchCount,
                        int equalKeySeriesCount,
                        boolean atLeastOneNonMatch,
                        boolean inputSelectedInUse,
                        int inputLogicalSize,
                        int spillCount,
                        int hashMapResultCount)
                 throws IOException,
                        HiveException

Generate the outer join output results for one vectorized row batch.

Parameters:

batch - The big table batch with any matching and any non matching rows both as selected in use.

allMatchCount - Number of matches in allMatchs.

equalKeySeriesCount - Number of single value matches.

atLeastOneNonMatch - Whether at least one row was a non-match.

inputSelectedInUse - A copy of the batch's selectedInUse flag on input to the process method.

inputLogicalSize - The batch's size on input to the process method.

spillCount - Number of spills in spills.

hashMapResultCount - Number of entries in hashMapResults.

Throws:

IOException

HiveException

generateOuterNulls

protected void generateOuterNulls(VectorizedRowBatch batch,
                                  int[] noMatchs,
                                  int noMatchSize)
                           throws IOException,
                                  HiveException

Generate the non matching outer join output results for one vectorized row batch. For each non matching row specified by parameter, generate nulls for the small table results.

Parameters:

batch - The big table batch with any matching and any non matching rows both as selected in use.

noMatchs - A subset of the rows of the batch that are non matches.

noMatchSize - Number of non matches in noMatchs.

Throws:

IOException

HiveException

finishOuterRepeated

public void finishOuterRepeated(VectorizedRowBatch batch,
                                JoinUtil.JoinResult joinResult,
                                VectorMapJoinHashMapResult hashMapResult,
                                boolean someRowsFilteredOut,
                                boolean inputSelectedInUse,
                                int inputLogicalSize)
                         throws IOException,
                                HiveException

Generate the outer join output results for one vectorized row batch with a repeated key. Any filter expressions will apply now since hash map lookup for outer join is complete.

Parameters:

batch - The big table batch with any matching and any non matching rows both as selected in use.

joinResult - The hash map lookup result for the repeated key.

hashMapResult - The array of all hash map results for the batch.

someRowsFilteredOut - Whether some rows of the repeated key batch were knocked out by the filter.

inputSelectedInUse - A copy of the batch's selectedInUse flag on input to the process method.

inputLogicalSize - The batch's size on input to the process method.

Throws:

IOException

HiveException

generateOuterNullsRepeatedAll

protected void generateOuterNullsRepeatedAll(VectorizedRowBatch batch)
                                      throws HiveException

Generate the non-match outer join output results for the whole repeating vectorized row batch. Each row will get nulls for all small table values.

Parameters:

batch - The big table batch.

Throws:

HiveException

generateFullOuterSmallTableNoMatches

protected void generateFullOuterSmallTableNoMatches(byte smallTablePos,
                                                    MapJoinTableContainer substituteSmallTable)
                                             throws HiveException

Overrides:

generateFullOuterSmallTableNoMatches in class MapJoinOperator

Throws:

HiveException

generateFullOuterLongKeySmallTableNoMatches

protected void generateFullOuterLongKeySmallTableNoMatches()
                                                    throws HiveException

Throws:

HiveException

generateFullOuterMultiKeySmallTableNoMatches

protected void generateFullOuterMultiKeySmallTableNoMatches()
                                                     throws HiveException

Throws:

HiveException

generateFullOuterStringKeySmallTableNoMatches

protected void generateFullOuterStringKeySmallTableNoMatches()
                                                      throws HiveException

Throws:

HiveException

fullOuterHashTableSetup

protected void fullOuterHashTableSetup()

fullOuterIntersectHashTableSetup

protected void fullOuterIntersectHashTableSetup()