org.apache.hadoop.hive.ql.exec.vector.mapjoin

Class VectorMapJoinCommonOperator

java.lang.Object

org.apache.hadoop.hive.ql.exec.Operator<T>

org.apache.hadoop.hive.ql.exec.CommonJoinOperator<T>

org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator<MapJoinDesc>

org.apache.hadoop.hive.ql.exec.MapJoinOperator

org.apache.hadoop.hive.ql.exec.vector.mapjoin.VectorMapJoinCommonOperator

All Implemented Interfaces:

Serializable, Cloneable, VectorizationContextRegion, VectorizationOperator, Node

Direct Known Subclasses:

VectorMapJoinGenerateResultOperator

public abstract class VectorMapJoinCommonOperator
extends MapJoinOperator
implements VectorizationOperator, VectorizationContextRegion

This class is common operator class for native vectorized map join. It contain common initialization logic. It is used by both inner and outer joins.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.exec.Operator

Operator.Counter, Operator.OperatorFunc, Operator.State

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean[]	allSmallTableKeyColumnIncluded
protected int[]	allSmallTableKeyColumnNums
protected long	batchCounter
protected int[]	bigTableByteColumnVectorColumns
protected VectorExpression[]	bigTableFilterExpressions
protected int[]	bigTableKeyColumnMap
protected String[]	bigTableKeyColumnNames
protected VectorExpression[]	bigTableKeyExpressions
protected TypeInfo[]	bigTableKeyTypeInfos
protected int[]	bigTableRetainColumnMap
protected VectorCopyRow	bigTableRetainedVectorCopy
protected TypeInfo[]	bigTableRetainTypeInfos
protected int[]	bigTableValueColumnMap
protected String[]	bigTableValueColumnNames
protected VectorExpression[]	bigTableValueExpressions
protected TypeInfo[]	bigTableValueTypeInfos
protected VectorColumnSourceMapping	fullOuterSmallTableKeyMapping
protected VectorMapJoinDesc.HashTableKeyType	hashTableKeyType
protected VectorMapJoinDesc.HashTableKind	hashTableKind
protected boolean	isOuterJoin
protected String	loggingPrefix
protected boolean	needCommonSetup
protected boolean	needFirstBatchSetup
protected boolean	needHashTableSetup
protected int[]	nonOuterSmallTableKeyByteColumnVectorColumns
protected int[]	nonOuterSmallTableKeyColumnMap
protected TypeInfo[]	nonOuterSmallTableKeyTypeInfos
protected VectorCopyRow	nonOuterSmallTableKeyVectorCopy
protected int[]	outerSmallTableKeyByteColumnVectorColumns
protected int[]	outerSmallTableKeyColumnMap
protected VectorColumnOutputMapping	outerSmallTableKeyMapping
protected VectorCopyRow	outerSmallTableKeyVectorCopy
protected int[]	outputProjection
protected TypeInfo[]	outputTypeInfos
protected VectorizedRowBatch	overflowBatch
protected int	overflowRepeatedThreshold
protected byte	posSingleVectorMapJoinSmallTable
protected VectorColumnSourceMapping	projectionMapping
protected long	rowCounter
protected int[]	smallTableByteColumnVectorColumns
protected VectorDeserializeRow<BinarySortableDeserializeRead>	smallTableKeyOuterVectorDeserializeRow
protected int[]	smallTableValueColumnMap
protected VectorColumnSourceMapping	smallTableValueMapping
protected VectorDeserializeRow<LazyBinaryDeserializeRead>	smallTableValueVectorDeserializeRow
protected VectorizedRowBatch	spillReplayBatch
protected boolean	useOverflowRepeatedThreshold
protected VectorizationContext	vContext
protected VectorMapJoinDesc	vectorDesc
protected VectorMapJoinHashTable	vectorMapJoinHashTable
protected VectorMapJoinInfo	vectorMapJoinInfo
protected VectorMapJoinDesc.VectorMapJoinVariation	vectorMapJoinVariation
protected VectorizationContext	vOutContext

Fields inherited from class org.apache.hadoop.hive.ql.exec.MapJoinOperator

firstSmallTable, fullOuterBigTableRetainSize, hashMapRowGetters, isFullOuterMapJoin, isTestingNoHashTableLoad, loader, mapJoinTables, matchTracker, spilledMapJoinTables

Fields inherited from class org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator

emptyList, joinKeys, joinKeysObjectInspectors, posBigTable

Fields inherited from class org.apache.hadoop.hive.ql.exec.CommonJoinOperator

alias, aliasFilterTags, aliasFilterTagsNext, condn, countAfterReport, dummyObj, dummyObjVectors, filterMaps, filterTags, forwardCache, heartbeatInterval, intermediate, joinFilterObjectInspectors, joinFilters, joinValues, joinValuesObjectInspectors, joinValuesStandardObjectInspectors, needsPostEvaluation, noOuterJoin, NOTSKIPBIGTABLE, nullsafes, numAliases, offsets, order, residualJoinFilters, residualJoinFiltersOIs, rowContainerPostFilteredOuterJoin, rowContainerStandardObjectInspectors, skipVectors, spillTableDesc, totalSz

Fields inherited from class org.apache.hadoop.hive.ql.exec.Operator

abortOp, asyncInitOperations, cContext, childOperators, childOperatorsArray, childOperatorsTag, conf, CONTEXT_NAME_KEY, done, groupKeyObject, HIVE_COUNTER_CREATED_DYNAMIC_PARTITIONS, HIVE_COUNTER_CREATED_FILES, HIVE_COUNTER_FATAL, id, inputObjInspectors, numRows, operatorId, out, outputObjInspector, parentOperators, reporter, runTimeNumRows, state, statsMap

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	VectorMapJoinCommonOperator() Kryo ctor.
	VectorMapJoinCommonOperator(CompilationOpContext ctx)
	VectorMapJoinCommonOperator(CompilationOpContext ctx, OperatorDesc conf, VectorizationContext vContext, VectorDesc vectorDesc)

Method Summary

Modifier and Type	Method and Description
protected void	commonSetup()
protected void	completeInitializationOp(Object[] os) This method can be used to retrieve the results from async operations started at init time - before the operator pipeline is started.
protected void	determineCommonInfo(boolean isOuter)
protected void	displayBatchColumns(VectorizedRowBatch batch, String batchName)
void	firstBatchSetup(VectorizedRowBatch batch)
protected HashTableLoader	getHashTableLoader(org.apache.hadoop.conf.Configuration hconf) This override lets us substitute our own fast vectorized hash table loader.
VectorizationContext	getInputVectorizationContext()
protected abstract String	getLoggingPrefix()
protected String	getLoggingPrefix(String className)
VectorizationContext	getOutputVectorizationContext()
OperatorType	getType() Return the type of the specific operator among the types in OperatorType.
VectorDesc	getVectorDesc()
void	hashTableSetup()
protected void	initializeOp(org.apache.hadoop.conf.Configuration hconf) Operator specific initialization.
protected void	initLoggingPrefix(String className)
void	process(Object row, int tag) Process the row.
abstract void	processBatch(VectorizedRowBatch batch)
void	setTestMapJoinTableContainer(int posSmallTable, MapJoinTableContainer testMapJoinTableContainer, MapJoinTableContainerSerDe mapJoinTableContainerSerDe)
protected VectorizedRowBatch	setupOverflowBatch()
protected void	setupVOutContext(List<String> outputColumnNames) Setup the vectorized row batch description of the output of the native vectorized map join operator.

Methods inherited from class org.apache.hadoop.hive.ql.exec.MapJoinOperator

canSkipJoinProcessing, canSkipReload, cleanUpInputFileChangedOp, closeOp, endGroup, generateFullOuterSmallTableNoMatches, generateMapMetaData, getCacheKey, getName, getOperatorName, getRefKey, getValueObjectInspectors, isInputFileChangeSensitive, loadHashTable, reloadHashTable, reProcessBigTable, setMapJoinKey, setMapJoinKeyNoNulls, spillBigTableRow, startGroup

Methods inherited from class org.apache.hadoop.hive.ql.exec.CommonJoinOperator

addToAliasFilterTags, checkAndGenObject, getFilteredValue, getFilterTag, getNextSize, getPosToAliasMap, hasFilter, internalForward, opAllowedAfterMapJoin, opAllowedBeforeMapJoin, reportProgress, setPosToAliasMap

Methods inherited from class org.apache.hadoop.hive.ql.exec.Operator

abort, acceptLimitPushdown, allInitializedParentsAreClosed, areAllParentsInitialized, augmentPlan, batchNeedsClone, cleanUpInputFileChanged, clone, cloneOp, cloneRecursiveChildren, close, columnNamesRowResolvedCanBeObtained, createDummy, defaultEndGroup, defaultStartGroup, dump, dump, flush, flushRecursive, forward, getAdditionalCounters, getChildOperators, getChildren, getColumnExprMap, getCompilationOpContext, getConf, getConfiguration, getCounterName, getDone, getExecContext, getGroupKeyObject, getIdentifier, getInputObjInspectors, getIsReduceSink, getMarker, getNextCntr, getNumChild, getNumParent, getOperatorId, getOpTraits, getOutputObjInspector, getParentOperators, getReduceOutputName, getSchema, getStatistics, getStats, initEvaluators, initEvaluators, initEvaluatorsAndReturnStruct, initialize, initialize, initializeChildren, initializeLocalWork, initOperatorId, isUseBucketizedHiveInputFormat, jobClose, jobCloseOp, logicalEquals, logicalEqualsTree, logStats, opAllowedBeforeSortMergeJoin, opAllowedConvertMapJoin, passExecContext, preorderMap, processGroup, removeChild, removeChildAndAdoptItsChildren, removeParent, removeParents, replaceChild, replaceParent, replaceTabAlias, reset, setAlias, setChildOperators, setColumnExprMap, setCompilationOpContext, setConf, setDone, setExecContext, setGroupKeyObject, setInputContext, setInputObjInspectors, setMarker, setNextVectorBatchGroupStatus, setOpTraits, setOutputCollector, setParentOperators, setReporter, setSchema, setStatistics, setUseBucketizedHiveInputFormat, supportAutomaticSortMergeJoin, supportSkewJoinOptimization, supportUnionRemoveOptimization, toString, toString, toString, vectorForward

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

loggingPrefix

protected transient String loggingPrefix

vectorDesc

protected VectorMapJoinDesc vectorDesc

vectorMapJoinInfo

protected VectorMapJoinInfo vectorMapJoinInfo

isOuterJoin

protected boolean isOuterJoin

posSingleVectorMapJoinSmallTable

protected byte posSingleVectorMapJoinSmallTable

vContext

protected VectorizationContext vContext

vOutContext

protected VectorizationContext vOutContext

vectorMapJoinVariation

protected VectorMapJoinDesc.VectorMapJoinVariation vectorMapJoinVariation

hashTableKind

protected VectorMapJoinDesc.HashTableKind hashTableKind

hashTableKeyType

protected VectorMapJoinDesc.HashTableKeyType hashTableKeyType

outputProjection

protected int[] outputProjection

outputTypeInfos

protected TypeInfo[] outputTypeInfos

bigTableFilterExpressions

protected VectorExpression[] bigTableFilterExpressions

bigTableKeyExpressions

protected VectorExpression[] bigTableKeyExpressions

bigTableValueExpressions

protected VectorExpression[] bigTableValueExpressions

bigTableKeyColumnMap

protected int[] bigTableKeyColumnMap

bigTableKeyColumnNames

protected String[] bigTableKeyColumnNames

bigTableKeyTypeInfos

protected TypeInfo[] bigTableKeyTypeInfos

bigTableValueColumnMap

protected int[] bigTableValueColumnMap

bigTableValueColumnNames

protected String[] bigTableValueColumnNames

bigTableValueTypeInfos

protected TypeInfo[] bigTableValueTypeInfos

bigTableRetainColumnMap

protected int[] bigTableRetainColumnMap

bigTableRetainTypeInfos

protected TypeInfo[] bigTableRetainTypeInfos

nonOuterSmallTableKeyColumnMap

protected int[] nonOuterSmallTableKeyColumnMap

nonOuterSmallTableKeyTypeInfos

protected TypeInfo[] nonOuterSmallTableKeyTypeInfos

outerSmallTableKeyMapping

protected VectorColumnOutputMapping outerSmallTableKeyMapping

fullOuterSmallTableKeyMapping

protected VectorColumnSourceMapping fullOuterSmallTableKeyMapping

smallTableValueMapping

protected VectorColumnSourceMapping smallTableValueMapping

projectionMapping

protected VectorColumnSourceMapping projectionMapping

outerSmallTableKeyColumnMap

protected int[] outerSmallTableKeyColumnMap

smallTableValueColumnMap

protected int[] smallTableValueColumnMap

bigTableByteColumnVectorColumns

protected int[] bigTableByteColumnVectorColumns

nonOuterSmallTableKeyByteColumnVectorColumns

protected int[] nonOuterSmallTableKeyByteColumnVectorColumns

outerSmallTableKeyByteColumnVectorColumns

protected int[] outerSmallTableKeyByteColumnVectorColumns

smallTableByteColumnVectorColumns

protected int[] smallTableByteColumnVectorColumns

useOverflowRepeatedThreshold

protected transient boolean useOverflowRepeatedThreshold

overflowRepeatedThreshold

protected transient int overflowRepeatedThreshold

bigTableRetainedVectorCopy

protected transient VectorCopyRow bigTableRetainedVectorCopy

allSmallTableKeyColumnNums

protected int[] allSmallTableKeyColumnNums

allSmallTableKeyColumnIncluded

protected boolean[] allSmallTableKeyColumnIncluded

smallTableKeyOuterVectorDeserializeRow

protected transient VectorDeserializeRow<BinarySortableDeserializeRead> smallTableKeyOuterVectorDeserializeRow

nonOuterSmallTableKeyVectorCopy

protected transient VectorCopyRow nonOuterSmallTableKeyVectorCopy

outerSmallTableKeyVectorCopy

protected transient VectorCopyRow outerSmallTableKeyVectorCopy

smallTableValueVectorDeserializeRow

protected transient VectorDeserializeRow<LazyBinaryDeserializeRead> smallTableValueVectorDeserializeRow

overflowBatch

protected transient VectorizedRowBatch overflowBatch

spillReplayBatch

protected transient VectorizedRowBatch spillReplayBatch

needCommonSetup

protected transient boolean needCommonSetup

needFirstBatchSetup

protected transient boolean needFirstBatchSetup

needHashTableSetup

protected transient boolean needHashTableSetup

vectorMapJoinHashTable

protected transient VectorMapJoinHashTable vectorMapJoinHashTable

batchCounter

protected transient long batchCounter

rowCounter

protected transient long rowCounter

Constructor Detail

VectorMapJoinCommonOperator

protected VectorMapJoinCommonOperator()

Kryo ctor.

VectorMapJoinCommonOperator

public VectorMapJoinCommonOperator(CompilationOpContext ctx)

VectorMapJoinCommonOperator

public VectorMapJoinCommonOperator(CompilationOpContext ctx,
                                   OperatorDesc conf,
                                   VectorizationContext vContext,
                                   VectorDesc vectorDesc)
                            throws HiveException

Throws:

HiveException

Method Detail

getLoggingPrefix

protected abstract String getLoggingPrefix()

getLoggingPrefix

protected String getLoggingPrefix(String className)

initLoggingPrefix

protected void initLoggingPrefix(String className)

determineCommonInfo

protected void determineCommonInfo(boolean isOuter)
                            throws HiveException

Throws:

HiveException

setupVOutContext

protected void setupVOutContext(List<String> outputColumnNames)

Setup the vectorized row batch description of the output of the native vectorized map join operator. Use the output projection we previously built from a mixture of input big table columns and new scratch columns.

getHashTableLoader

protected HashTableLoader getHashTableLoader(org.apache.hadoop.conf.Configuration hconf)

This override lets us substitute our own fast vectorized hash table loader.

Overrides:

getHashTableLoader in class MapJoinOperator

initializeOp

protected void initializeOp(org.apache.hadoop.conf.Configuration hconf)
                     throws HiveException

Description copied from class: Operator

Operator specific initialization.

Overrides:

initializeOp in class MapJoinOperator

Throws:

HiveException

completeInitializationOp

protected void completeInitializationOp(Object[] os)
                                 throws HiveException

Description copied from class: Operator

This method can be used to retrieve the results from async operations started at init time - before the operator pipeline is started.

Overrides:

completeInitializationOp in class MapJoinOperator

Throws:

HiveException

setTestMapJoinTableContainer

public void setTestMapJoinTableContainer(int posSmallTable,
                                         MapJoinTableContainer testMapJoinTableContainer,
                                         MapJoinTableContainerSerDe mapJoinTableContainerSerDe)

Overrides:

setTestMapJoinTableContainer in class MapJoinOperator

setupOverflowBatch

protected VectorizedRowBatch setupOverflowBatch()
                                         throws HiveException

Throws:

HiveException

commonSetup

protected void commonSetup()
                    throws HiveException

Throws:

HiveException

firstBatchSetup

public void firstBatchSetup(VectorizedRowBatch batch)
                     throws HiveException

Throws:

HiveException

hashTableSetup

public void hashTableSetup()
                    throws HiveException

Throws:

HiveException

processBatch

public abstract void processBatch(VectorizedRowBatch batch)
                           throws HiveException

Throws:

HiveException

process

public void process(Object row,
                    int tag)
             throws HiveException

Description copied from class: Operator

Process the row.

Overrides:

process in class MapJoinOperator

Parameters:

row - The object representing the row.

tag - The tag of the row usually means which parent this row comes from. Rows with the same tag should have exactly the same rowInspector all the time.

Throws:

HiveException

displayBatchColumns

protected void displayBatchColumns(VectorizedRowBatch batch,
                                   String batchName)

getType

public OperatorType getType()

Description copied from class: Operator

Return the type of the specific operator among the types in OperatorType.

Overrides:

getType in class MapJoinOperator

Returns:

OperatorType.*

getInputVectorizationContext

public VectorizationContext getInputVectorizationContext()

Specified by:

getInputVectorizationContext in interface VectorizationOperator

getVectorDesc

public VectorDesc getVectorDesc()

Specified by:

getVectorDesc in interface VectorizationOperator

getOutputVectorizationContext

public VectorizationContext getOutputVectorizationContext()

Specified by:

getOutputVectorizationContext in interface VectorizationContextRegion