org.apache.beam.runners.dataflow

Class DataflowPipelineJob

java.lang.Object

org.apache.beam.runners.dataflow.DataflowPipelineJob

All Implemented Interfaces:

org.apache.beam.sdk.PipelineResult

Direct Known Subclasses:

DataflowTemplateJob

public class DataflowPipelineJob
extends java.lang.Object
implements org.apache.beam.sdk.PipelineResult

A DataflowPipelineJob represents a job submitted to Dataflow using DataflowRunner.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.beam.sdk.PipelineResult

org.apache.beam.sdk.PipelineResult.State

Field Summary

Fields

Modifier and Type	Field and Description
protected java.lang.String	jobId The id for the job.
protected static org.apache.beam.sdk.util.FluentBackoff	STATUS_BACKOFF_FACTORY
protected org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.BiMap<org.apache.beam.sdk.runners.AppliedPTransform<?,?,?>,java.lang.String>	transformStepNames

Constructor Summary

Constructors

Constructor and Description
DataflowPipelineJob(DataflowClient dataflowClient, java.lang.String jobId, DataflowPipelineOptions dataflowOptions, java.util.Map<org.apache.beam.sdk.runners.AppliedPTransform<?,?,?>,java.lang.String> transformStepNames) Constructs the job.

Method Summary

Modifier and Type	Method and Description
org.apache.beam.sdk.PipelineResult.State	cancel()
DataflowPipelineOptions	getDataflowOptions()
java.lang.String	getJobId() Get the id of this job.
java.lang.String	getProjectId() Get the project this job exists in.
java.lang.String	getRegion() Get the region this job exists in.
DataflowPipelineJob	getReplacedByJob() Returns a new DataflowPipelineJob for the job that replaced this one, if applicable.
org.apache.beam.sdk.PipelineResult.State	getState()
org.apache.beam.sdk.metrics.MetricResults	metrics()
org.apache.beam.sdk.PipelineResult.State	waitUntilFinish()
org.apache.beam.sdk.PipelineResult.State	waitUntilFinish(org.joda.time.Duration duration)
org.apache.beam.sdk.PipelineResult.State	waitUntilFinish(org.joda.time.Duration duration, MonitoringUtil.JobMessagesHandler messageHandler) Waits until the pipeline finishes and returns the final status.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

jobId

protected java.lang.String jobId

The id for the job.

transformStepNames

protected org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.BiMap<org.apache.beam.sdk.runners.AppliedPTransform<?,?,?>,java.lang.String> transformStepNames

STATUS_BACKOFF_FACTORY

protected static final org.apache.beam.sdk.util.FluentBackoff STATUS_BACKOFF_FACTORY

Constructor Detail

DataflowPipelineJob

public DataflowPipelineJob(DataflowClient dataflowClient,
                           java.lang.String jobId,
                           DataflowPipelineOptions dataflowOptions,
                           java.util.Map<org.apache.beam.sdk.runners.AppliedPTransform<?,?,?>,java.lang.String> transformStepNames)

Constructs the job.

Parameters:

jobId - the job id

dataflowOptions - used to configure the client for the Dataflow Service

transformStepNames - a mapping from AppliedPTransforms to Step Names

Method Detail

getJobId

public java.lang.String getJobId()

Get the id of this job.

getProjectId

public java.lang.String getProjectId()

Get the project this job exists in.

getDataflowOptions

public DataflowPipelineOptions getDataflowOptions()

getRegion

public java.lang.String getRegion()

Get the region this job exists in.

getReplacedByJob

public DataflowPipelineJob getReplacedByJob()

Returns a new DataflowPipelineJob for the job that replaced this one, if applicable.

Throws:

java.lang.IllegalStateException - if called before the job has terminated or if the job terminated but was not updated

waitUntilFinish

@Nullable
public org.apache.beam.sdk.PipelineResult.State waitUntilFinish()

Specified by:

waitUntilFinish in interface org.apache.beam.sdk.PipelineResult

waitUntilFinish

@Nullable
public org.apache.beam.sdk.PipelineResult.State waitUntilFinish(org.joda.time.Duration duration)

Specified by:

waitUntilFinish in interface org.apache.beam.sdk.PipelineResult

waitUntilFinish

@Nullable
public org.apache.beam.sdk.PipelineResult.State waitUntilFinish(org.joda.time.Duration duration,
                                                                          MonitoringUtil.JobMessagesHandler messageHandler)
                                                                   throws java.io.IOException,
                                                                          java.lang.InterruptedException

Waits until the pipeline finishes and returns the final status.

Parameters:

duration - The time to wait for the job to finish. Provide a value less than 1 ms for an infinite wait.

messageHandler - If non null this handler will be invoked for each batch of messages received.

Returns:

The final state of the job or null on timeout or if the thread is interrupted.

Throws:

java.io.IOException - If there is a persistent problem getting job information.

java.lang.InterruptedException

cancel

public org.apache.beam.sdk.PipelineResult.State cancel()
                                                throws java.io.IOException

Specified by:

cancel in interface org.apache.beam.sdk.PipelineResult

Throws:

java.io.IOException

getState

public org.apache.beam.sdk.PipelineResult.State getState()

Specified by:

getState in interface org.apache.beam.sdk.PipelineResult

metrics

public org.apache.beam.sdk.metrics.MetricResults metrics()

Specified by:

metrics in interface org.apache.beam.sdk.PipelineResult