org.apache.hadoop.hive.ql.exec.mr

Class MapRedTask

java.lang.Object

org.apache.hadoop.hive.ql.exec.Task<MapredWork>

org.apache.hadoop.hive.ql.exec.mr.ExecDriver

org.apache.hadoop.hive.ql.exec.mr.MapRedTask

All Implemented Interfaces:

Serializable, HadoopJobExecHook, Node

public class MapRedTask
extends ExecDriver
implements Serializable

Extension of ExecDriver: - can optionally spawn a map-reduce task from a separate jvm - will make last minute adjustments to map-reduce job parameters, viz: * estimating number of reducers * estimating whether job should run locally

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.exec.Task

Task.FeedType, Task.TaskState

Field Summary

Fields inherited from class org.apache.hadoop.hive.ql.exec.mr.ExecDriver

job, jobExecHelper, LOG, memoryMXBean

Fields inherited from class org.apache.hadoop.hive.ql.exec.Task

BACKUP_COMMON_JOIN, backupChildrenTasks, backupTask, childTasks, clonedConf, COMMON_JOIN, conf, console, context, CONVERTED_MAPJOIN, CONVERTED_MAPJOIN_LOCAL, CONVERTED_SORTMERGEJOIN, deferredWorkContext, feedSubscribers, HINTED_MAPJOIN, HINTED_MAPJOIN_LOCAL, id, jobID, MAPJOIN_ONLY_NOBACKUP, NO_TAG, parentTasks, queryDisplay, queryPlan, queryState, rootTask, taskCounters, taskHandle, taskQueue, taskTag, work

Constructor Summary

Constructors

Constructor and Description
MapRedTask()

Method Summary

Modifier and Type	Method and Description
int	execute() Execute a query plan using Hadoop.
Operator<? extends OperatorDesc>	getReducer(MapWork mapWork)
static String	isEligibleForLocalMode(HiveConf conf, int numReducers, long inputLength, long inputFileCount) Find out if a job can be run in local mode based on it's characteristics
boolean	mapDone()
boolean	mapStarted()
boolean	reduceDone()
boolean	reduceStarted()
void	shutdown()
void	updateTaskMetrics(org.apache.hadoop.hive.common.metrics.common.Metrics metrics) Provide metrics on the type and number of tasks executed by the HiveServer
void	updateWebUiStats(MapRedStats mapRedStats, org.apache.hadoop.mapred.RunningJob rj)

Methods inherited from class org.apache.hadoop.hive.ql.exec.mr.ExecDriver

checkFatalErrors, generateCmdLine, getExternalHandle, getMapWork, getName, getResource, getTopOperators, getType, hasReduce, initialize, isMapRedTask, isTaskShutdown, logPlanProgress, main, propagateSplitSettings, requireLock, setInputAttributes

Methods inherited from class org.apache.hadoop.hive.ql.exec.Task

addDependentTask, canExecuteInParallel, cloneConf, done, equals, executeTask, findLeafs, getAndInitBackupTask, getBackupChildrenTasks, getBackupTask, getChildren, getChildTasks, getConf, getContext, getCounters, getDeferredWorkContext, getDependentTasks, getDiagnosticsMessage, getException, getFeedSubscribers, getHive, getId, getInitialized, getJobID, getNumChild, getNumParent, getParentTasks, getQueryPlan, getQueryState, getQueued, getStatusMessage, getTaskHandle, getTaskQueue, getTaskState, getTaskTag, getTxnMgr, getWork, hashCode, isFetchSource, isLocalMode, isMapRedLocalTask, isNotInitialized, isRootTask, isRunnable, pushFeed, receiveFeed, removeDependentTask, removeFromChildrenTasks, setBackupChildrenTasks, setBackupTask, setChildTasks, setConf, setConsole, setDeferredWorkContext, setDiagnosticMessage, setDone, setException, setFeedSubscribers, setFetchSource, setId, setInitialized, setLocalMode, setParentTasks, setQueryDisplay, setQueryPlan, setQueued, setRootTask, setStarted, setState, setStatusMessage, setTaskQueue, setTaskTag, setWork, started, subscribeFeed, toString, updateStatusInQueryDisplay

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

MapRedTask

public MapRedTask()

Method Detail

execute

public int execute()

Description copied from class: ExecDriver

Execute a query plan using Hadoop.

Overrides:

execute in class ExecDriver

Returns:

status of executing the task

mapStarted

public boolean mapStarted()

Overrides:

mapStarted in class ExecDriver

reduceStarted

public boolean reduceStarted()

Overrides:

reduceStarted in class ExecDriver

mapDone

public boolean mapDone()

Overrides:

mapDone in class ExecDriver

reduceDone

public boolean reduceDone()

Overrides:

reduceDone in class ExecDriver

updateTaskMetrics

public void updateTaskMetrics(org.apache.hadoop.hive.common.metrics.common.Metrics metrics)

Description copied from class: Task

Provide metrics on the type and number of tasks executed by the HiveServer

Overrides:

updateTaskMetrics in class Task<MapredWork>

isEligibleForLocalMode

public static String isEligibleForLocalMode(HiveConf conf,
                                            int numReducers,
                                            long inputLength,
                                            long inputFileCount)

Find out if a job can be run in local mode based on it's characteristics

Parameters:

conf - Hive Configuration

numReducers - total number of reducers for this job

inputLength - the size of the input

inputFileCount - the number of files of input

Returns:

String null if job is eligible for local mode, reason otherwise

getReducer

public Operator<? extends OperatorDesc> getReducer(MapWork mapWork)

Overrides:

getReducer in class Task<MapredWork>

updateWebUiStats

public void updateWebUiStats(MapRedStats mapRedStats,
                             org.apache.hadoop.mapred.RunningJob rj)

shutdown

public void shutdown()

Overrides:

shutdown in class ExecDriver