org.apache.tez.dag.app

Class TezTaskCommunicatorImpl

java.lang.Object

org.apache.tez.serviceplugins.api.TaskCommunicator

org.apache.tez.dag.app.TezTaskCommunicatorImpl

All Implemented Interfaces:

org.apache.tez.common.ServicePluginLifecycle

Direct Known Subclasses:

TezLocalTaskCommunicatorImpl

@InterfaceAudience.Private
public class TezTaskCommunicatorImpl
extends TaskCommunicator

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TezTaskCommunicatorImpl.ContainerInfo

Field Summary

Fields

Modifier and Type	Field and Description
protected InetSocketAddress	address
protected ConcurrentMap<org.apache.tez.dag.records.TezTaskAttemptID,org.apache.hadoop.yarn.api.records.ContainerId>	attemptToContainerMap
protected org.apache.hadoop.conf.Configuration	conf
protected ConcurrentMap<org.apache.hadoop.yarn.api.records.ContainerId,TezTaskCommunicatorImpl.ContainerInfo>	registeredContainers
protected org.apache.hadoop.ipc.Server	server
protected org.apache.hadoop.security.token.Token<org.apache.tez.common.security.JobTokenIdentifier>	sessionToken
protected String	tokenIdentifier

Constructor Summary

Constructors

Constructor and Description
TezTaskCommunicatorImpl(TaskCommunicatorContext taskCommunicatorContext) Construct the service.

Method Summary

Modifier and Type	Method and Description
void	dagComplete(int dagIdentifier) Indicates the current running dag is complete.
InetSocketAddress	getAddress() Return the address, if any, that the service listens on
protected org.apache.hadoop.conf.Configuration	getConf()
protected org.apache.hadoop.yarn.api.records.ContainerId	getContainerForAttempt(org.apache.tez.dag.records.TezTaskAttemptID taskAttemptId)
protected TezTaskCommunicatorImpl.ContainerInfo	getContainerInfo(org.apache.hadoop.yarn.api.records.ContainerId containerId)
Object	getMetaInfo() Share meta-information such as host:port information where the Task Communicator may be listening.
protected org.apache.hadoop.security.token.Token<org.apache.tez.common.security.JobTokenIdentifier>	getSessionToken()
protected String	getTokenIdentifier()
org.apache.tez.common.TezTaskUmbilicalProtocol	getUmbilical()
void	onVertexStateUpdated(org.apache.tez.dag.api.event.VertexStateUpdate stateUpdate) Receive notifications on vertex state changes.
void	registerContainerEnd(org.apache.hadoop.yarn.api.records.ContainerId containerId, org.apache.tez.serviceplugins.api.ContainerEndReason endReason, String diagnostics) Register the end of a container.
void	registerRunningContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId, String host, int port) Register a new container.
void	registerRunningTaskAttempt(org.apache.hadoop.yarn.api.records.ContainerId containerId, org.apache.tez.runtime.api.impl.TaskSpec taskSpec, Map<String,org.apache.hadoop.yarn.api.records.LocalResource> additionalResources, org.apache.hadoop.security.Credentials credentials, boolean credentialsChanged, int priority) Register a task attempt to execute on a container
void	shutdown() Stop the service.
void	start() An entry point for starting the service.
protected void	startRpcServer()
protected void	stopRpcServer()
void	unregisterRunningTaskAttempt(org.apache.tez.dag.records.TezTaskAttemptID taskAttemptID, org.apache.tez.serviceplugins.api.TaskAttemptEndReason endReason, String diagnostics) Register the completion of a task.

Methods inherited from class org.apache.tez.serviceplugins.api.TaskCommunicator

getCompletedLogsUrl, getContext, getInProgressLogsUrl, initialize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

registeredContainers

protected final ConcurrentMap<org.apache.hadoop.yarn.api.records.ContainerId,TezTaskCommunicatorImpl.ContainerInfo> registeredContainers

attemptToContainerMap

protected final ConcurrentMap<org.apache.tez.dag.records.TezTaskAttemptID,org.apache.hadoop.yarn.api.records.ContainerId> attemptToContainerMap

tokenIdentifier

protected final String tokenIdentifier

sessionToken

protected final org.apache.hadoop.security.token.Token<org.apache.tez.common.security.JobTokenIdentifier> sessionToken

conf

protected final org.apache.hadoop.conf.Configuration conf

address

protected InetSocketAddress address

server

protected volatile org.apache.hadoop.ipc.Server server

Constructor Detail

TezTaskCommunicatorImpl

public TezTaskCommunicatorImpl(TaskCommunicatorContext taskCommunicatorContext)

Construct the service.

Method Detail

start

public void start()

Description copied from class: TaskCommunicator

An entry point for starting the service. Order of service setup. Constructor, initialize(), start() - when starting a service.

Specified by:

start in interface org.apache.tez.common.ServicePluginLifecycle

Overrides:

start in class TaskCommunicator

shutdown

public void shutdown()

Description copied from class: TaskCommunicator

Stop the service. This could be invoked at any point, when the service is no longer required - including in case of errors.

Specified by:

shutdown in interface org.apache.tez.common.ServicePluginLifecycle

Overrides:

shutdown in class TaskCommunicator

startRpcServer

protected void startRpcServer()

stopRpcServer

protected void stopRpcServer()

getConf

protected org.apache.hadoop.conf.Configuration getConf()

registerRunningContainer

public void registerRunningContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                                     String host,
                                     int port)

Description copied from class: TaskCommunicator

Register a new container.

Specified by:

registerRunningContainer in class TaskCommunicator

Parameters:

containerId - the associated containerId

host - the hostname on which the container runs

port - the port for the service which is running the container

registerContainerEnd

public void registerContainerEnd(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                                 org.apache.tez.serviceplugins.api.ContainerEndReason endReason,
                                 String diagnostics)

Description copied from class: TaskCommunicator

Register the end of a container. This can be caused by preemption, the container completing successfully, etc.

Specified by:

registerContainerEnd in class TaskCommunicator

Parameters:

containerId - the associated containerId

endReason - the end reason for the container completing

diagnostics - diagnostics associated with the container end

registerRunningTaskAttempt

public void registerRunningTaskAttempt(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                                       org.apache.tez.runtime.api.impl.TaskSpec taskSpec,
                                       Map<String,org.apache.hadoop.yarn.api.records.LocalResource> additionalResources,
                                       org.apache.hadoop.security.Credentials credentials,
                                       boolean credentialsChanged,
                                       int priority)

Description copied from class: TaskCommunicator

Register a task attempt to execute on a container

Specified by:

registerRunningTaskAttempt in class TaskCommunicator

Parameters:

containerId - the containerId on which this task needs to run

taskSpec - the task specifications for the task to be executed

additionalResources - additional local resources which may be required to run this task on the container

credentials - the credentials required to run this task

credentialsChanged - whether the credentials are different from the original credentials associated with this container

priority - the priority of the task being executed

unregisterRunningTaskAttempt

public void unregisterRunningTaskAttempt(org.apache.tez.dag.records.TezTaskAttemptID taskAttemptID,
                                         org.apache.tez.serviceplugins.api.TaskAttemptEndReason endReason,
                                         String diagnostics)

Description copied from class: TaskCommunicator

Register the completion of a task. This may be a result of preemption, the container dying, the node dying, the task completing to success

Specified by:

unregisterRunningTaskAttempt in class TaskCommunicator

Parameters:

taskAttemptID - the task attempt which has completed / needs to be completed

endReason - the endReason for the task attempt.

diagnostics - diagnostics associated with the task end

getAddress

public InetSocketAddress getAddress()

Description copied from class: TaskCommunicator

Return the address, if any, that the service listens on

Specified by:

getAddress in class TaskCommunicator

Returns:

the address

onVertexStateUpdated

public void onVertexStateUpdated(org.apache.tez.dag.api.event.VertexStateUpdate stateUpdate)

Description copied from class: TaskCommunicator

Receive notifications on vertex state changes.

State changes will be received based on the registration via InputInitializerContext.registerForVertexStateUpdates(String, java.util.Set). Notifications will be received for all registered state changes, and not just for the latest state update. They will be in order in which the state change occurred.

Extensive processing should not be performed via this method call. Instead this should just be used as a notification mechanism.
This method may be invoked concurrently with other invocations into the TaskCommunicator and multi-threading/concurrency implications must be considered.

Specified by:

onVertexStateUpdated in class TaskCommunicator

Parameters:

stateUpdate - an event indicating the name of the vertex, and it's updated state. Additional information may be available for specific events, Look at the type hierarchy for VertexStateUpdate

dagComplete

public void dagComplete(int dagIdentifier)

Description copied from class: TaskCommunicator

Indicates the current running dag is complete. The TaskCommunicatorContext can be used to query information about the current dag during the duration of the dagComplete invocation.

After this, the contents returned from querying the context may change at any point - due to the next dag being submitted.

Specified by:

dagComplete in class TaskCommunicator

Parameters:

dagIdentifier - the unique numerical identifier for the DAG in the specified execution context.

getMetaInfo

public Object getMetaInfo()

Description copied from class: TaskCommunicator

Share meta-information such as host:port information where the Task Communicator may be listening. Primarily for use by compatible launchers to learn this information.

Specified by:

getMetaInfo in class TaskCommunicator

Returns:

meta info for the task communicator

getTokenIdentifier

protected String getTokenIdentifier()

getSessionToken

protected org.apache.hadoop.security.token.Token<org.apache.tez.common.security.JobTokenIdentifier> getSessionToken()

getUmbilical

public org.apache.tez.common.TezTaskUmbilicalProtocol getUmbilical()

getContainerInfo

protected TezTaskCommunicatorImpl.ContainerInfo getContainerInfo(org.apache.hadoop.yarn.api.records.ContainerId containerId)

getContainerForAttempt

protected org.apache.hadoop.yarn.api.records.ContainerId getContainerForAttempt(org.apache.tez.dag.records.TezTaskAttemptID taskAttemptId)