org.apache.samza.job.yarn

Class YarnClusterResourceManager

java.lang.Object

org.apache.samza.clustermanager.ClusterResourceManager

org.apache.samza.job.yarn.YarnClusterResourceManager

All Implemented Interfaces:

org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler, org.apache.hadoop.yarn.client.api.async.NMClientAsync.CallbackHandler

public class YarnClusterResourceManager
extends org.apache.samza.clustermanager.ClusterResourceManager
implements org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler, org.apache.hadoop.yarn.client.api.async.NMClientAsync.CallbackHandler

An YarnClusterResourceManager implements a ClusterResourceManager using Yarn as the underlying resource manager. This class is as an adaptor between Yarn and translates Yarn callbacks into Samza specific callback methods as specified in Callback. Thread-safety: 1.Start and stop methods should NOT be called from multiple threads. 2.ALL callbacks from the YarnContainerManager are invoked from a single Callback thread of the AMRMClient. 3.Stop should not be called more than once.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.samza.clustermanager.ClusterResourceManager

org.apache.samza.clustermanager.ClusterResourceManager.Callback

Field Summary

Fields inherited from class org.apache.samza.clustermanager.ClusterResourceManager

clusterManagerCallback

Constructor Summary

Constructors

Constructor and Description
YarnClusterResourceManager(org.apache.samza.config.Config config, org.apache.samza.coordinator.JobModelManager jobModelManager, org.apache.samza.clustermanager.ClusterResourceManager.Callback callback, org.apache.samza.clustermanager.SamzaApplicationState samzaAppState) Creates an YarnClusterResourceManager from config, a jobModelReader and a callback.

Method Summary

Modifier and Type	Method and Description
void	cancelResourceRequest(org.apache.samza.clustermanager.SamzaResourceRequest request) Remove a previously submitted resource request.
float	getProgress()
boolean	isResourceExpired(org.apache.samza.clustermanager.SamzaResource resource)
void	launchStreamProcessor(org.apache.samza.clustermanager.SamzaResource resource, org.apache.samza.job.CommandBuilder builder) Requests the launch of a StreamProcessor with the specified ID on the resource
void	onContainersAllocated(java.util.List<org.apache.hadoop.yarn.api.records.Container> containers) Callback invoked from Yarn when containers are allocated.
void	onContainersCompleted(java.util.List<org.apache.hadoop.yarn.api.records.ContainerStatus> statuses) Callback invoked from Yarn when containers complete.
void	onContainerStarted(org.apache.hadoop.yarn.api.records.ContainerId containerId, java.util.Map<java.lang.String,java.nio.ByteBuffer> allServiceResponse)
void	onContainerStatusReceived(org.apache.hadoop.yarn.api.records.ContainerId containerId, org.apache.hadoop.yarn.api.records.ContainerStatus containerStatus)
void	onContainerStopped(org.apache.hadoop.yarn.api.records.ContainerId containerId)
void	onError(java.lang.Throwable e) Callback invoked when there is an error in the Yarn client.
void	onGetContainerStatusError(org.apache.hadoop.yarn.api.records.ContainerId containerId, java.lang.Throwable t)
void	onNodesUpdated(java.util.List<org.apache.hadoop.yarn.api.records.NodeReport> updatedNodes)
void	onShutdownRequest()
void	onStartContainerError(org.apache.hadoop.yarn.api.records.ContainerId containerId, java.lang.Throwable t)
void	onStopContainerError(org.apache.hadoop.yarn.api.records.ContainerId containerId, java.lang.Throwable t)
void	releaseResources(org.apache.samza.clustermanager.SamzaResource resource) Requests the YarnContainerManager to release a resource.
void	requestResources(org.apache.samza.clustermanager.SamzaResourceRequest resourceRequest) Request resources for running container processes.
void	runProcessor(java.lang.String processorId, org.apache.hadoop.yarn.api.records.Container container, org.apache.samza.job.CommandBuilder cmdBuilder) Runs a process as specified by the command builder on the container.
void	start() Starts the YarnClusterResourceManager and initialize all its sub-systems.
void	stop(org.apache.samza.clustermanager.SamzaApplicationState.SamzaAppStatus status) Stops the YarnContainerManager and all its sub-components.
void	stopStreamProcessor(org.apache.samza.clustermanager.SamzaResource resource)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

YarnClusterResourceManager

public YarnClusterResourceManager(org.apache.samza.config.Config config,
                                  org.apache.samza.coordinator.JobModelManager jobModelManager,
                                  org.apache.samza.clustermanager.ClusterResourceManager.Callback callback,
                                  org.apache.samza.clustermanager.SamzaApplicationState samzaAppState)

Creates an YarnClusterResourceManager from config, a jobModelReader and a callback.

Parameters:

config - to instantiate the cluster manager with

jobModelManager - the jobModel manager to get the job model (mostly for the UI)

callback - the callback to receive events from Yarn.

samzaAppState - samza app state for display in the UI

Method Detail

start

public void start()

Starts the YarnClusterResourceManager and initialize all its sub-systems. Attempting to start an already started cluster manager will return immediately.

Specified by:

start in class org.apache.samza.clustermanager.ClusterResourceManager

requestResources

public void requestResources(org.apache.samza.clustermanager.SamzaResourceRequest resourceRequest)

Request resources for running container processes.

Specified by:

requestResources in class org.apache.samza.clustermanager.ClusterResourceManager

releaseResources

public void releaseResources(org.apache.samza.clustermanager.SamzaResource resource)

Requests the YarnContainerManager to release a resource. If the app cannot use the resource or wants to give up the resource, it can release them.

Specified by:

releaseResources in class org.apache.samza.clustermanager.ClusterResourceManager

Parameters:

resource - to be released

launchStreamProcessor

public void launchStreamProcessor(org.apache.samza.clustermanager.SamzaResource resource,
                                  org.apache.samza.job.CommandBuilder builder)

Requests the launch of a StreamProcessor with the specified ID on the resource

Specified by:

launchStreamProcessor in class org.apache.samza.clustermanager.ClusterResourceManager

Parameters:

resource - the SamzaResource on which to launch the StreamProcessor

builder - the builder to build the resource launch command from TODO: Support non-builder methods to launch resources. Maybe, refactor into a ContainerLaunchStrategy interface

stopStreamProcessor

public void stopStreamProcessor(org.apache.samza.clustermanager.SamzaResource resource)

Specified by:

stopStreamProcessor in class org.apache.samza.clustermanager.ClusterResourceManager

cancelResourceRequest

public void cancelResourceRequest(org.apache.samza.clustermanager.SamzaResourceRequest request)

Remove a previously submitted resource request. The previous container request may have been submitted. Even after the remove request, a Callback implementation must be prepared to receive an allocation for the previous request. This is merely a best effort cancellation.

Specified by:

cancelResourceRequest in class org.apache.samza.clustermanager.ClusterResourceManager

Parameters:

request - the request to be cancelled

stop

public void stop(org.apache.samza.clustermanager.SamzaApplicationState.SamzaAppStatus status)

Stops the YarnContainerManager and all its sub-components. Stop should NOT be called from multiple threads. TODO: fix this to make stop idempotent?.

Specified by:

stop in class org.apache.samza.clustermanager.ClusterResourceManager

onContainersCompleted

public void onContainersCompleted(java.util.List<org.apache.hadoop.yarn.api.records.ContainerStatus> statuses)

Callback invoked from Yarn when containers complete. This translates the yarn callbacks into Samza specific ones.

Specified by:

onContainersCompleted in interface org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler

Parameters:

statuses - the YarnContainerStatus callbacks from Yarn.

onContainersAllocated

public void onContainersAllocated(java.util.List<org.apache.hadoop.yarn.api.records.Container> containers)

Callback invoked from Yarn when containers are allocated. This translates the yarn callbacks into Samza specific ones.

Specified by:

onContainersAllocated in interface org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler

Parameters:

containers - the list of Container returned by Yarn.

onShutdownRequest

public void onShutdownRequest()

Specified by:

onShutdownRequest in interface org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler

onNodesUpdated

public void onNodesUpdated(java.util.List<org.apache.hadoop.yarn.api.records.NodeReport> updatedNodes)

Specified by:

onNodesUpdated in interface org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler

getProgress

public float getProgress()

Specified by:

getProgress in interface org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler

onError

public void onError(java.lang.Throwable e)

Callback invoked when there is an error in the Yarn client. This delegates the callback handling to the ClusterResourceManager.Callback instance.

Specified by:

onError in interface org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler

onContainerStarted

public void onContainerStarted(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                               java.util.Map<java.lang.String,java.nio.ByteBuffer> allServiceResponse)

Specified by:

onContainerStarted in interface org.apache.hadoop.yarn.client.api.async.NMClientAsync.CallbackHandler

onContainerStatusReceived

public void onContainerStatusReceived(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                                      org.apache.hadoop.yarn.api.records.ContainerStatus containerStatus)

Specified by:

onContainerStatusReceived in interface org.apache.hadoop.yarn.client.api.async.NMClientAsync.CallbackHandler

onContainerStopped

public void onContainerStopped(org.apache.hadoop.yarn.api.records.ContainerId containerId)

Specified by:

onContainerStopped in interface org.apache.hadoop.yarn.client.api.async.NMClientAsync.CallbackHandler

onStartContainerError

public void onStartContainerError(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                                  java.lang.Throwable t)

Specified by:

onStartContainerError in interface org.apache.hadoop.yarn.client.api.async.NMClientAsync.CallbackHandler

onGetContainerStatusError

public void onGetContainerStatusError(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                                      java.lang.Throwable t)

Specified by:

onGetContainerStatusError in interface org.apache.hadoop.yarn.client.api.async.NMClientAsync.CallbackHandler

onStopContainerError

public void onStopContainerError(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                                 java.lang.Throwable t)

Specified by:

onStopContainerError in interface org.apache.hadoop.yarn.client.api.async.NMClientAsync.CallbackHandler

isResourceExpired

public boolean isResourceExpired(org.apache.samza.clustermanager.SamzaResource resource)

Overrides:

isResourceExpired in class org.apache.samza.clustermanager.ClusterResourceManager

runProcessor

public void runProcessor(java.lang.String processorId,
                         org.apache.hadoop.yarn.api.records.Container container,
                         org.apache.samza.job.CommandBuilder cmdBuilder)
                  throws java.io.IOException

Runs a process as specified by the command builder on the container.

Parameters:

processorId - id of the samza processor to run (passed as a command line parameter to the process)

container - the yarn container to run the processor on.

cmdBuilder - the command builder that encapsulates the command, and the context

Throws:

java.io.IOException - on IO exceptions running the container