Package io.trino.execution.scheduler

Class PipelinedStageExecution

java.lang.Object

io.trino.execution.scheduler.PipelinedStageExecution

All Implemented Interfaces:

StageExecution

public class PipelinedStageExecution
extends Object
implements StageExecution

This class is designed to facilitate the pipelined mode of execution.

In the pipeline mode the tasks are executed in all-or-nothing fashion with all the intermediate data being "piped" between stages in a streaming way.

This class has two main responsibilities:

1. Linking pipelined stages together. If a new task is scheduled the implementation notifies upstream stages to add an additional output buffer for the task as well as it notifies the downstream stage to update a list of source tasks. It is also responsible of notifying both upstream and downstream stages when no more tasks will be added.

2. Facilitates state transitioning for a pipelined stage execution according to the all-or-noting model. If any of the tasks fail the implementation is responsible for terminating all remaining tasks as well as propagating the original error. If all the tasks finish successfully the implementation is responsible for notifying the scheduler about a successful completion of a given stage.

Nested Class Summary

Nested classes/interfaces inherited from interface io.trino.execution.scheduler.StageExecution

StageExecution.State

Method Summary

Modifier and Type	Method	Description
void	abort()
void	addCompletedDriverGroupsChangedListener(Consumer<Set<Lifespan>> newlyCompletedDriverGroupConsumer)
void	addStateChangeListener(StateMachine.StateChangeListener<StageExecution.State> stateChangeListener)	Listener is always notified asynchronously using a dedicated notification thread pool so, care should be taken to avoid leaking this when adding a listener in a constructor.
void	beginScheduling()
void	cancel()
static PipelinedStageExecution	createPipelinedStageExecution(SqlStage stage, Map<PlanFragmentId,io.trino.execution.scheduler.OutputBufferManager> outputBufferManagers, TaskLifecycleListener taskLifecycleListener, FailureDetector failureDetector, Executor executor, Optional<int[]> bucketToPartition, int attempt)
void	fail(Throwable failureCause)
void	failTask(TaskId taskId, Throwable failureCause)
List<RemoteTask>	getAllTasks()
int	getAttemptId()
Optional<ExecutionFailureInfo>	getFailureCause()
PlanFragment	getFragment()
StageId	getStageId()
StageExecution.State	getState()
TaskLifecycleListener	getTaskLifecycleListener()
List<TaskStatus>	getTaskStatuses()
boolean	isAnyTaskBlocked()
void	recordGetSplitTime(long start)
Optional<RemoteTask>	scheduleTask(InternalNode node, int partition, com.google.common.collect.Multimap<PlanNodeId,Split> initialSplits, com.google.common.collect.Multimap<PlanNodeId,Lifespan> noMoreSplitsForLifespan)
void	schedulingComplete()
void	schedulingComplete(PlanNodeId partitionedSource)
void	transitionToSchedulingSplits()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

createPipelinedStageExecution

public static PipelinedStageExecution createPipelinedStageExecution(SqlStage stage,
                                                                    Map<PlanFragmentId,io.trino.execution.scheduler.OutputBufferManager> outputBufferManagers,
                                                                    TaskLifecycleListener taskLifecycleListener,
                                                                    FailureDetector failureDetector,
                                                                    Executor executor,
                                                                    Optional<int[]> bucketToPartition,
                                                                    int attempt)

getState

public StageExecution.State getState()

Specified by:

getState in interface StageExecution

addStateChangeListener

public void addStateChangeListener(StateMachine.StateChangeListener<StageExecution.State> stateChangeListener)

Listener is always notified asynchronously using a dedicated notification thread pool so, care should be taken to avoid leaking this when adding a listener in a constructor.

Specified by:

addStateChangeListener in interface StageExecution

addCompletedDriverGroupsChangedListener

public void addCompletedDriverGroupsChangedListener(Consumer<Set<Lifespan>> newlyCompletedDriverGroupConsumer)

Specified by:

addCompletedDriverGroupsChangedListener in interface StageExecution

beginScheduling

public void beginScheduling()

Specified by:

beginScheduling in interface StageExecution

transitionToSchedulingSplits

public void transitionToSchedulingSplits()

Specified by:

transitionToSchedulingSplits in interface StageExecution

schedulingComplete

public void schedulingComplete()

Specified by:

schedulingComplete in interface StageExecution

schedulingComplete

public void schedulingComplete(PlanNodeId partitionedSource)

Specified by:

schedulingComplete in interface StageExecution

cancel

public void cancel()

Specified by:

cancel in interface StageExecution

abort

public void abort()

Specified by:

abort in interface StageExecution

fail

public void fail(Throwable failureCause)

failTask

public void failTask(TaskId taskId,
                     Throwable failureCause)

Specified by:

failTask in interface StageExecution

scheduleTask

public Optional<RemoteTask> scheduleTask(InternalNode node,
                                         int partition,
                                         com.google.common.collect.Multimap<PlanNodeId,Split> initialSplits,
                                         com.google.common.collect.Multimap<PlanNodeId,Lifespan> noMoreSplitsForLifespan)

Specified by:

scheduleTask in interface StageExecution

getTaskLifecycleListener

public TaskLifecycleListener getTaskLifecycleListener()

Specified by:

getTaskLifecycleListener in interface StageExecution

getAllTasks

public List<RemoteTask> getAllTasks()

Specified by:

getAllTasks in interface StageExecution

getTaskStatuses

public List<TaskStatus> getTaskStatuses()

Specified by:

getTaskStatuses in interface StageExecution

isAnyTaskBlocked

public boolean isAnyTaskBlocked()

Specified by:

isAnyTaskBlocked in interface StageExecution

recordGetSplitTime

public void recordGetSplitTime(long start)

Specified by:

recordGetSplitTime in interface StageExecution

getStageId

public StageId getStageId()

Specified by:

getStageId in interface StageExecution

getAttemptId

public int getAttemptId()

Specified by:

getAttemptId in interface StageExecution

getFragment

public PlanFragment getFragment()

Specified by:

getFragment in interface StageExecution

getFailureCause

public Optional<ExecutionFailureInfo> getFailureCause()

Specified by:

getFailureCause in interface StageExecution