org.apache.hadoop.hive.ql.exec

Class MapJoinOperator

java.lang.Object

org.apache.hadoop.hive.ql.exec.Operator<T>

org.apache.hadoop.hive.ql.exec.CommonJoinOperator<T>

org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator<MapJoinDesc>

org.apache.hadoop.hive.ql.exec.MapJoinOperator

All Implemented Interfaces:

Serializable, Cloneable, Node

Direct Known Subclasses:

VectorMapJoinBaseOperator, VectorMapJoinCommonOperator

public class MapJoinOperator
extends AbstractMapJoinOperator<MapJoinDesc>
implements Serializable

Map side Join operator implementation.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.exec.Operator

Operator.Counter, Operator.OperatorFunc, Operator.State

Field Summary

Fields

Modifier and Type	Field and Description
protected HybridHashTableContainer	firstSmallTable
protected int	fullOuterBigTableRetainSize
protected MapJoinTableContainer.ReusableGetAdaptor[]	hashMapRowGetters
protected boolean	isFullOuterMapJoin
protected boolean	isTestingNoHashTableLoad
protected HashTableLoader	loader
protected MapJoinTableContainer[]	mapJoinTables
protected MatchTracker	matchTracker
protected MapJoinBytesTableContainer[]	spilledMapJoinTables

Fields inherited from class org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator

emptyList, joinKeys, joinKeysObjectInspectors, posBigTable

Fields inherited from class org.apache.hadoop.hive.ql.exec.CommonJoinOperator

alias, aliasFilterTags, aliasFilterTagsNext, condn, countAfterReport, dummyObj, dummyObjVectors, filterMaps, filterTags, forwardCache, heartbeatInterval, intermediate, joinFilterObjectInspectors, joinFilters, joinValues, joinValuesObjectInspectors, joinValuesStandardObjectInspectors, needsPostEvaluation, noOuterJoin, NOTSKIPBIGTABLE, nullsafes, numAliases, offsets, order, residualJoinFilters, residualJoinFiltersOIs, rowContainerPostFilteredOuterJoin, rowContainerStandardObjectInspectors, skipVectors, spillTableDesc, totalSz

Fields inherited from class org.apache.hadoop.hive.ql.exec.Operator

abortOp, asyncInitOperations, cContext, childOperators, childOperatorsArray, childOperatorsTag, conf, CONTEXT_NAME_KEY, done, groupKeyObject, HIVE_COUNTER_CREATED_DYNAMIC_PARTITIONS, HIVE_COUNTER_CREATED_FILES, HIVE_COUNTER_FATAL, id, inputObjInspectors, numRows, operatorId, out, outputObjInspector, parentOperators, reporter, runTimeNumRows, state, statsMap

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	MapJoinOperator() Kryo ctor.
	MapJoinOperator(AbstractMapJoinOperator<? extends MapJoinDesc> mjop)
	MapJoinOperator(CompilationOpContext ctx)

Method Summary

Modifier and Type	Method and Description
protected boolean	canSkipJoinProcessing(ExecMapperContext mapContext)
protected boolean	canSkipReload(ExecMapperContext mapContext)
void	cleanUpInputFileChangedOp()
void	closeOp(boolean abort) All done.
protected void	completeInitializationOp(Object[] os) This method can be used to retrieve the results from async operations started at init time - before the operator pipeline is started.
void	endGroup() Forward a record of join results.
protected void	generateFullOuterSmallTableNoMatches(byte smallTablePos, MapJoinTableContainer substituteSmallTable)
void	generateMapMetaData()
String	getCacheKey()
protected HashTableLoader	getHashTableLoader(org.apache.hadoop.conf.Configuration hconf)
String	getName() Implements the getName function for the Node Interface.
static String	getOperatorName()
protected MapJoinKey	getRefKey(byte alias)
OperatorType	getType() Return the type of the specific operator among the types in OperatorType.
protected List<ObjectInspector>	getValueObjectInspectors(byte alias, List<ObjectInspector>[] aliasToObjectInspectors)
protected void	initializeOp(org.apache.hadoop.conf.Configuration hconf) Operator specific initialization.
protected boolean	isInputFileChangeSensitive(ExecMapperContext mapContext)
protected org.apache.commons.lang3.tuple.Pair<MapJoinTableContainer[],MapJoinTableContainerSerDe[]>	loadHashTable(ExecMapperContext mapContext, MapredContext mrContext)
void	process(Object row, int tag) Process the row.
protected void	reloadHashTable(byte pos, int partitionId) Reload hashtable from the hash partition.
protected void	reProcessBigTable(int partitionId) Iterate over the big table row container and feed process() with leftover rows
protected JoinUtil.JoinResult	setMapJoinKey(MapJoinTableContainer.ReusableGetAdaptor dest, Object row, byte alias)
protected JoinUtil.JoinResult	setMapJoinKeyNoNulls(MapJoinTableContainer.ReusableGetAdaptor dest, Object row, byte alias, MatchTracker matchTracker)
void	setTestMapJoinTableContainer(int posSmallTable, MapJoinTableContainer testMapJoinTableContainer, MapJoinTableContainerSerDe mapJoinTableContainerSerDe)
protected void	spillBigTableRow(MapJoinTableContainer hybridHtContainer, Object row) Postpone processing the big table row temporarily by spilling it to a row container
void	startGroup()

Methods inherited from class org.apache.hadoop.hive.ql.exec.CommonJoinOperator

addToAliasFilterTags, checkAndGenObject, getFilteredValue, getFilterTag, getNextSize, getPosToAliasMap, hasFilter, internalForward, opAllowedAfterMapJoin, opAllowedBeforeMapJoin, reportProgress, setPosToAliasMap

Methods inherited from class org.apache.hadoop.hive.ql.exec.Operator

abort, acceptLimitPushdown, allInitializedParentsAreClosed, areAllParentsInitialized, augmentPlan, cleanUpInputFileChanged, clone, cloneOp, cloneRecursiveChildren, close, columnNamesRowResolvedCanBeObtained, createDummy, defaultEndGroup, defaultStartGroup, dump, dump, flush, flushRecursive, forward, getAdditionalCounters, getChildOperators, getChildren, getColumnExprMap, getCompilationOpContext, getConf, getConfiguration, getCounterName, getDone, getExecContext, getGroupKeyObject, getIdentifier, getInputObjInspectors, getIsReduceSink, getMarker, getNextCntr, getNumChild, getNumParent, getOperatorId, getOpTraits, getOutputObjInspector, getParentOperators, getReduceOutputName, getSchema, getStatistics, getStats, initEvaluators, initEvaluators, initEvaluatorsAndReturnStruct, initialize, initialize, initializeChildren, initializeLocalWork, initOperatorId, isUseBucketizedHiveInputFormat, jobClose, jobCloseOp, logicalEquals, logicalEqualsTree, logStats, opAllowedBeforeSortMergeJoin, opAllowedConvertMapJoin, passExecContext, preorderMap, processGroup, removeChild, removeChildAndAdoptItsChildren, removeParent, removeParents, replaceChild, replaceParent, replaceTabAlias, reset, setAlias, setChildOperators, setColumnExprMap, setCompilationOpContext, setConf, setDone, setExecContext, setGroupKeyObject, setInputContext, setInputObjInspectors, setMarker, setNextVectorBatchGroupStatus, setOpTraits, setOutputCollector, setParentOperators, setReporter, setSchema, setStatistics, setUseBucketizedHiveInputFormat, supportAutomaticSortMergeJoin, supportSkewJoinOptimization, supportUnionRemoveOptimization, toString, toString, toString, vectorForward

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

loader

protected HashTableLoader loader

mapJoinTables

protected transient MapJoinTableContainer[] mapJoinTables

hashMapRowGetters

protected transient MapJoinTableContainer.ReusableGetAdaptor[] hashMapRowGetters

spilledMapJoinTables

protected transient MapJoinBytesTableContainer[] spilledMapJoinTables

firstSmallTable

protected HybridHashTableContainer firstSmallTable

isFullOuterMapJoin

protected transient boolean isFullOuterMapJoin

fullOuterBigTableRetainSize

protected transient int fullOuterBigTableRetainSize

matchTracker

protected transient MatchTracker matchTracker

isTestingNoHashTableLoad

protected transient boolean isTestingNoHashTableLoad

Constructor Detail

MapJoinOperator

protected MapJoinOperator()

Kryo ctor.

MapJoinOperator

public MapJoinOperator(CompilationOpContext ctx)

MapJoinOperator

public MapJoinOperator(AbstractMapJoinOperator<? extends MapJoinDesc> mjop)

Method Detail

endGroup

public void endGroup()
              throws HiveException

Description copied from class: CommonJoinOperator

Forward a record of join results.

Overrides:

endGroup in class CommonJoinOperator<MapJoinDesc>

Throws:

HiveException

startGroup

public void startGroup()
                throws HiveException

Overrides:

startGroup in class CommonJoinOperator<MapJoinDesc>

Throws:

HiveException

getHashTableLoader

protected HashTableLoader getHashTableLoader(org.apache.hadoop.conf.Configuration hconf)

getCacheKey

public String getCacheKey()

initializeOp

protected void initializeOp(org.apache.hadoop.conf.Configuration hconf)
                     throws HiveException

Description copied from class: Operator

Operator specific initialization.

Overrides:

initializeOp in class AbstractMapJoinOperator<MapJoinDesc>

Throws:

HiveException

completeInitializationOp

protected void completeInitializationOp(Object[] os)
                                 throws HiveException

Description copied from class: Operator

This method can be used to retrieve the results from async operations started at init time - before the operator pipeline is started.

Overrides:

completeInitializationOp in class Operator<MapJoinDesc>

Throws:

HiveException

setTestMapJoinTableContainer

public void setTestMapJoinTableContainer(int posSmallTable,
                                         MapJoinTableContainer testMapJoinTableContainer,
                                         MapJoinTableContainerSerDe mapJoinTableContainerSerDe)

getValueObjectInspectors

protected List<ObjectInspector> getValueObjectInspectors(byte alias,
                                                         List<ObjectInspector>[] aliasToObjectInspectors)

Overrides:

getValueObjectInspectors in class AbstractMapJoinOperator<MapJoinDesc>

generateMapMetaData

public void generateMapMetaData()
                         throws HiveException

Throws:

HiveException

loadHashTable

protected org.apache.commons.lang3.tuple.Pair<MapJoinTableContainer[],MapJoinTableContainerSerDe[]> loadHashTable(ExecMapperContext mapContext,
                                                                                                                  MapredContext mrContext)
                                                                                                           throws HiveException

Throws:

HiveException

cleanUpInputFileChangedOp

public void cleanUpInputFileChangedOp()
                               throws HiveException

Overrides:

cleanUpInputFileChangedOp in class Operator<MapJoinDesc>

Throws:

HiveException

setMapJoinKey

protected JoinUtil.JoinResult setMapJoinKey(MapJoinTableContainer.ReusableGetAdaptor dest,
                                            Object row,
                                            byte alias)
                                     throws HiveException

Throws:

HiveException

setMapJoinKeyNoNulls

protected JoinUtil.JoinResult setMapJoinKeyNoNulls(MapJoinTableContainer.ReusableGetAdaptor dest,
                                                   Object row,
                                                   byte alias,
                                                   MatchTracker matchTracker)
                                            throws HiveException

Throws:

HiveException

getRefKey

protected MapJoinKey getRefKey(byte alias)

process

public void process(Object row,
                    int tag)
             throws HiveException

Description copied from class: Operator

Process the row.

Specified by:

process in class Operator<MapJoinDesc>

Parameters:

row - The object representing the row.

tag - The tag of the row usually means which parent this row comes from. Rows with the same tag should have exactly the same rowInspector all the time.

Throws:

HiveException

spillBigTableRow

protected void spillBigTableRow(MapJoinTableContainer hybridHtContainer,
                                Object row)
                         throws HiveException

Postpone processing the big table row temporarily by spilling it to a row container

Parameters:

hybridHtContainer - Hybrid hashtable container

row - big table row

Throws:

HiveException

generateFullOuterSmallTableNoMatches

protected void generateFullOuterSmallTableNoMatches(byte smallTablePos,
                                                    MapJoinTableContainer substituteSmallTable)
                                             throws HiveException

Throws:

HiveException

closeOp

public void closeOp(boolean abort)
             throws HiveException

Description copied from class: CommonJoinOperator

All done.

Overrides:

closeOp in class AbstractMapJoinOperator<MapJoinDesc>

Throws:

HiveException

reloadHashTable

protected void reloadHashTable(byte pos,
                               int partitionId)
                        throws IOException,
                               HiveException,
                               SerDeException,
                               ClassNotFoundException

Reload hashtable from the hash partition. It can have two steps: 1) Deserialize a serialized hash table, and 2) Merge every key/value pair from small table container into the hash table

Parameters:

pos - position of small table

partitionId - the partition of the small table to be reloaded from

Throws:

IOException

HiveException

SerDeException

ClassNotFoundException

reProcessBigTable

protected void reProcessBigTable(int partitionId)
                          throws HiveException

Iterate over the big table row container and feed process() with leftover rows

Parameters:

partitionId - the partition from which to take out spilled big table rows

Throws:

HiveException

getName

public String getName()

Implements the getName function for the Node Interface.

Specified by:

getName in interface Node

Overrides:

getName in class CommonJoinOperator<MapJoinDesc>

Returns:

the name of the operator

getOperatorName

public static String getOperatorName()

getType

public OperatorType getType()

Description copied from class: Operator

Return the type of the specific operator among the types in OperatorType.

Overrides:

getType in class AbstractMapJoinOperator<MapJoinDesc>

Returns:

OperatorType.*

isInputFileChangeSensitive

protected boolean isInputFileChangeSensitive(ExecMapperContext mapContext)

canSkipReload

protected boolean canSkipReload(ExecMapperContext mapContext)

canSkipJoinProcessing

protected boolean canSkipJoinProcessing(ExecMapperContext mapContext)