org.apache.beam.runners.dataflow

Class DataflowPipelineJob

java.lang.Object

org.apache.beam.runners.dataflow.DataflowPipelineJob

All Implemented Interfaces:

PipelineResult

Direct Known Subclasses:

DataflowTemplateJob

public class DataflowPipelineJob
extends Object
implements PipelineResult

A DataflowPipelineJob represents a job submitted to Dataflow using DataflowRunner.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.beam.sdk.PipelineResult

PipelineResult.State

Field Summary

Fields

Modifier and Type	Field and Description
protected String	jobId The id for the job.
protected static FluentBackoff	STATUS_BACKOFF_FACTORY
protected com.google.common.collect.BiMap<AppliedPTransform<?,?,?>,String>	transformStepNames

Constructor Summary

Constructors

Constructor and Description
DataflowPipelineJob(DataflowClient dataflowClient, String jobId, DataflowPipelineOptions dataflowOptions, Map<AppliedPTransform<?,?,?>,String> transformStepNames) Constructs the job.

Method Summary

Modifier and Type	Method and Description
PipelineResult.State	cancel()
DataflowPipelineOptions	getDataflowOptions()
String	getJobId() Get the id of this job.
String	getProjectId() Get the project this job exists in.
String	getRegion() Get the region this job exists in.
DataflowPipelineJob	getReplacedByJob() Returns a new DataflowPipelineJob for the job that replaced this one, if applicable.
PipelineResult.State	getState()
MetricResults	metrics()
PipelineResult.State	waitUntilFinish()
PipelineResult.State	waitUntilFinish(org.joda.time.Duration duration)
PipelineResult.State	waitUntilFinish(org.joda.time.Duration duration, MonitoringUtil.JobMessagesHandler messageHandler) Waits until the pipeline finishes and returns the final status.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

jobId

protected String jobId

The id for the job.

transformStepNames

protected com.google.common.collect.BiMap<AppliedPTransform<?,?,?>,String> transformStepNames

STATUS_BACKOFF_FACTORY

protected static final FluentBackoff STATUS_BACKOFF_FACTORY

Constructor Detail

DataflowPipelineJob

public DataflowPipelineJob(DataflowClient dataflowClient,
                           String jobId,
                           DataflowPipelineOptions dataflowOptions,
                           Map<AppliedPTransform<?,?,?>,String> transformStepNames)

Constructs the job.

Parameters:

jobId - the job id

dataflowOptions - used to configure the client for the Dataflow Service

transformStepNames - a mapping from AppliedPTransforms to Step Names

Method Detail

getJobId

public String getJobId()

Get the id of this job.

getProjectId

public String getProjectId()

Get the project this job exists in.

getDataflowOptions

public DataflowPipelineOptions getDataflowOptions()

getRegion

public String getRegion()

Get the region this job exists in.

getReplacedByJob

public DataflowPipelineJob getReplacedByJob()

Returns a new DataflowPipelineJob for the job that replaced this one, if applicable.

Throws:

IllegalStateException - if called before the job has terminated or if the job terminated but was not updated

waitUntilFinish

@Nullable
public PipelineResult.State waitUntilFinish()

Specified by:

waitUntilFinish in interface PipelineResult

waitUntilFinish

@Nullable
public PipelineResult.State waitUntilFinish(org.joda.time.Duration duration)

Specified by:

waitUntilFinish in interface PipelineResult

waitUntilFinish

@Nullable
public PipelineResult.State waitUntilFinish(org.joda.time.Duration duration,
                                                      MonitoringUtil.JobMessagesHandler messageHandler)
                                               throws IOException,
                                                      InterruptedException

Waits until the pipeline finishes and returns the final status.

Parameters:

duration - The time to wait for the job to finish. Provide a value less than 1 ms for an infinite wait.

messageHandler - If non null this handler will be invoked for each batch of messages received.

Returns:

The final state of the job or null on timeout or if the thread is interrupted.

Throws:

IOException - If there is a persistent problem getting job information.

InterruptedException

cancel

public PipelineResult.State cancel()
                            throws IOException

Specified by:

cancel in interface PipelineResult

Throws:

IOException

getState

public PipelineResult.State getState()

Specified by:

getState in interface PipelineResult

metrics

public MetricResults metrics()

Specified by:

metrics in interface PipelineResult