org.apache.tez.dag.app.dag.speculation.legacy

Class LegacySpeculator

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.tez.dag.app.dag.speculation.legacy.LegacySpeculator

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.service.Service

public class LegacySpeculator
extends org.apache.hadoop.service.AbstractService

Maintains runtime estimation statistics. Makes periodic updates estimates based on progress and decides on when to trigger a speculative attempt. Speculation attempts are triggered when the estimated runtime is more than a threshold beyond the mean runtime and the original task still has enough estimated runtime left that the speculative version is expected to finish sooner than that. If the original is close to completion then we dont start a speculation because it may be likely a wasted attempt. There is a delay between successive speculations.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

org.apache.hadoop.service.Service.STATE

Constructor Summary

Constructors

Constructor and Description
LegacySpeculator(org.apache.hadoop.conf.Configuration conf, AppContext context, Vertex vertex)
LegacySpeculator(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.yarn.util.Clock clock, Vertex vertex)
LegacySpeculator(org.apache.hadoop.conf.Configuration conf, TaskRuntimeEstimator estimator, org.apache.hadoop.yarn.util.Clock clock, Vertex vertex)

Method Summary

Modifier and Type	Method and Description
protected void	addSpeculativeAttempt(org.apache.tez.dag.records.TezTaskID taskID)
Runnable	createThread()
int	getMinimumAllowedSpeculativeTasks()
double	getProportionRunningTasksSpeculatable()
double	getProportionTotalTasksSpeculatable()
long	getSoonestRetryAfterNoSpeculate()
long	getSoonestRetryAfterSpeculate()
void	handle(SpeculatorEvent event)
boolean	isStarted()
void	notifyAttemptStarted(org.apache.tez.dag.records.TezTaskAttemptID taId, long timestamp)
void	notifyAttemptStatusUpdate(org.apache.tez.dag.records.TezTaskAttemptID taId, TaskAttemptState reportedState, long timestamp)
protected void	serviceStart()
protected void	serviceStop()

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, serviceInit, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

LegacySpeculator

public LegacySpeculator(org.apache.hadoop.conf.Configuration conf,
                        AppContext context,
                        Vertex vertex)

LegacySpeculator

public LegacySpeculator(org.apache.hadoop.conf.Configuration conf,
                        org.apache.hadoop.yarn.util.Clock clock,
                        Vertex vertex)

LegacySpeculator

public LegacySpeculator(org.apache.hadoop.conf.Configuration conf,
                        TaskRuntimeEstimator estimator,
                        org.apache.hadoop.yarn.util.Clock clock,
                        Vertex vertex)

Method Detail

getMinimumAllowedSpeculativeTasks

public int getMinimumAllowedSpeculativeTasks()

getProportionTotalTasksSpeculatable

public double getProportionTotalTasksSpeculatable()

getProportionRunningTasksSpeculatable

public double getProportionRunningTasksSpeculatable()

getSoonestRetryAfterNoSpeculate

public long getSoonestRetryAfterNoSpeculate()

getSoonestRetryAfterSpeculate

public long getSoonestRetryAfterSpeculate()

serviceStart

protected void serviceStart()
                     throws Exception

Overrides:

serviceStart in class org.apache.hadoop.service.AbstractService

Throws:

Exception

isStarted

public boolean isStarted()

serviceStop

protected void serviceStop()
                    throws Exception

Overrides:

serviceStop in class org.apache.hadoop.service.AbstractService

Throws:

Exception

createThread

public Runnable createThread()

notifyAttemptStarted

public void notifyAttemptStarted(org.apache.tez.dag.records.TezTaskAttemptID taId,
                                 long timestamp)

notifyAttemptStatusUpdate

public void notifyAttemptStatusUpdate(org.apache.tez.dag.records.TezTaskAttemptID taId,
                                      TaskAttemptState reportedState,
                                      long timestamp)

handle

public void handle(SpeculatorEvent event)

addSpeculativeAttempt

protected void addSpeculativeAttempt(org.apache.tez.dag.records.TezTaskID taskID)