org.apache.beam.runners.flink.translation.wrappers.streaming

Class ExecutableStageDoFnOperator<InputT,OutputT>

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<OutputT>

org.apache.beam.runners.flink.translation.wrappers.streaming.AbstractStreamOperatorCompat<org.apache.beam.sdk.util.WindowedValue<OutputT>>

org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator<InputT,OutputT>

org.apache.beam.runners.flink.translation.wrappers.streaming.ExecutableStageDoFnOperator<InputT,OutputT>

All Implemented Interfaces:

java.io.Serializable, org.apache.flink.runtime.state.CheckpointListener, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.OneInputStreamOperator<org.apache.beam.sdk.util.WindowedValue<InputT>,org.apache.beam.sdk.util.WindowedValue<OutputT>>, org.apache.flink.streaming.api.operators.SetupableStreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>, org.apache.flink.streaming.api.operators.StreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator, org.apache.flink.streaming.api.operators.Triggerable<java.nio.ByteBuffer,org.apache.beam.runners.core.TimerInternals.TimerData>, org.apache.flink.streaming.api.operators.TwoInputStreamOperator<org.apache.beam.sdk.util.WindowedValue<InputT>,org.apache.beam.sdk.transforms.join.RawUnionValue,org.apache.beam.sdk.util.WindowedValue<OutputT>>, org.apache.flink.util.Disposable

public class ExecutableStageDoFnOperator<InputT,OutputT>
extends DoFnOperator<InputT,OutputT>

This operator is the streaming equivalent of the FlinkExecutableStageFunction. It sends all received elements to the SDK harness and emits the received back elements to the downstream operators. It also takes care of handling side inputs and state.

TODO Integrate support for progress updates and metrics

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator

DoFnOperator.BufferedOutputManager<OutputT>, DoFnOperator.FlinkStepContext, DoFnOperator.MultiOutputOutputManagerFactory<OutputT>

Field Summary

Fields inherited from class org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator

additionalOutputTags, bufferingDoFnRunner, doFn, doFnRunner, keyCoder, keyedStateInternals, mainOutputTag, outputManager, outputManagerFactory, pushbackDoFnRunner, serializedOptions, sideInputHandler, sideInputReader, sideInputs, sideInputTagMapping, stepName, timerInternals, timerService, windowingStrategy

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

chainingStrategy, config, latencyStats, metrics, output, processingTimeService

Constructor Summary

Constructors

Constructor and Description

ExecutableStageDoFnOperator(java.lang.String stepName, org.apache.beam.sdk.coders.Coder<org.apache.beam.sdk.util.WindowedValue<InputT>> windowedInputCoder, java.util.Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.coders.Coder<?>> outputCoders, org.apache.beam.sdk.values.TupleTag<OutputT> mainOutputTag, java.util.List<org.apache.beam.sdk.values.TupleTag<?>> additionalOutputTags, org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.OutputManagerFactory<OutputT> outputManagerFactory, java.util.Map<java.lang.Integer,org.apache.beam.sdk.values.PCollectionView<?>> sideInputTagMapping, java.util.Collection<org.apache.beam.sdk.values.PCollectionView<?>> sideInputs, java.util.Map<org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload.SideInputId,org.apache.beam.sdk.values.PCollectionView<?>> sideInputIds, org.apache.beam.sdk.options.PipelineOptions options, org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload payload, org.apache.beam.runners.fnexecution.provisioning.JobInfo jobInfo, FlinkExecutableStageContextFactory contextFactory, java.util.Map<java.lang.String,org.apache.beam.sdk.values.TupleTag<?>> outputMap, org.apache.beam.sdk.values.WindowingStrategy windowingStrategy, org.apache.beam.sdk.coders.Coder keyCoder, org.apache.flink.api.java.functions.KeySelector<org.apache.beam.sdk.util.WindowedValue<InputT>,?> keySelector) Constructor.

Method Summary

Modifier and Type	Method and Description
protected void	addSideInputValue(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.beam.sdk.transforms.join.RawUnionValue> streamRecord) Add the side input value.
long	applyInputWatermarkHold(long inputWatermark) Allows to apply a hold to the input watermark.
long	applyOutputWatermarkHold(long currentOutputWatermark, long potentialOutputWatermark) Allows to apply a hold to the output watermark before it is send out.
void	cleanUp() Release all of the operator's resources.
protected org.apache.beam.runners.core.DoFnRunner<InputT,OutputT>	createWrappingDoFnRunner(org.apache.beam.runners.core.DoFnRunner<InputT,OutputT> wrappedRunner, org.apache.beam.runners.core.StepContext stepContext)
protected void	fireTimerInternal(java.nio.ByteBuffer key, org.apache.beam.runners.core.TimerInternals.TimerData timer)
void	flushData() Flush all remaining buffered data.
java.nio.ByteBuffer	getCurrentKey()
protected java.util.concurrent.locks.Lock	getLockToAcquireForStateAccessDuringBundles() Subclasses may provide a lock to ensure that the state backend is not accessed concurrently during bundle execution.
void	notifyCheckpointComplete(long checkpointId)
void	open()
void	setCurrentKey(java.lang.Object key) We don't want to set anything here.
void	setKeyContextElement1(org.apache.flink.streaming.runtime.streamrecord.StreamRecord record) Note: This is only relevant when we have a stateful DoFn.

Methods inherited from class org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator

fireTimer, getBundleFinalizer, getCurrentOutputWatermark, getDoFn, getEffectiveInputWatermark, initializeState, invokeFinishBundle, onEventTime, onProcessingTime, prepareSnapshotPreBarrier, processElement, processElement1, processElement2, processWatermark, processWatermark1, processWatermark2, setBundleFinishedCallback, setPreBundleCallback, setup, snapshotState

Methods inherited from class org.apache.beam.runners.flink.translation.wrappers.streaming.AbstractStreamOperatorCompat

close, dispose, getTimeServiceManagerCompat

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

getChainingStrategy, getContainingTask, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, numEventTimeTimers, numProcessingTimeTimers, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, reportOrForwardLatencyMarker, setChainingStrategy, setKeyContextElement2, setProcessingTimeService, snapshotState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

processLatencyMarker

Methods inherited from interface org.apache.flink.streaming.api.operators.TwoInputStreamOperator

processLatencyMarker1, processLatencyMarker2

Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

close, dispose, getMetricGroup, getOperatorID, initializeState, setKeyContextElement2, snapshotState

Methods inherited from interface org.apache.flink.runtime.state.CheckpointListener

notifyCheckpointAborted

Constructor Detail

ExecutableStageDoFnOperator

public ExecutableStageDoFnOperator(java.lang.String stepName,
                                   org.apache.beam.sdk.coders.Coder<org.apache.beam.sdk.util.WindowedValue<InputT>> windowedInputCoder,
                                   java.util.Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.coders.Coder<?>> outputCoders,
                                   org.apache.beam.sdk.values.TupleTag<OutputT> mainOutputTag,
                                   java.util.List<org.apache.beam.sdk.values.TupleTag<?>> additionalOutputTags,
                                   org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.OutputManagerFactory<OutputT> outputManagerFactory,
                                   java.util.Map<java.lang.Integer,org.apache.beam.sdk.values.PCollectionView<?>> sideInputTagMapping,
                                   java.util.Collection<org.apache.beam.sdk.values.PCollectionView<?>> sideInputs,
                                   java.util.Map<org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload.SideInputId,org.apache.beam.sdk.values.PCollectionView<?>> sideInputIds,
                                   org.apache.beam.sdk.options.PipelineOptions options,
                                   org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload payload,
                                   org.apache.beam.runners.fnexecution.provisioning.JobInfo jobInfo,
                                   FlinkExecutableStageContextFactory contextFactory,
                                   java.util.Map<java.lang.String,org.apache.beam.sdk.values.TupleTag<?>> outputMap,
                                   org.apache.beam.sdk.values.WindowingStrategy windowingStrategy,
                                   org.apache.beam.sdk.coders.Coder keyCoder,
                                   org.apache.flink.api.java.functions.KeySelector<org.apache.beam.sdk.util.WindowedValue<InputT>,?> keySelector)

Constructor.

Method Detail

getLockToAcquireForStateAccessDuringBundles

protected java.util.concurrent.locks.Lock getLockToAcquireForStateAccessDuringBundles()

Description copied from class: DoFnOperator

Subclasses may provide a lock to ensure that the state backend is not accessed concurrently during bundle execution.

Overrides:

getLockToAcquireForStateAccessDuringBundles in class DoFnOperator<InputT,OutputT>

open

public void open()
          throws java.lang.Exception

Specified by:

open in interface org.apache.flink.streaming.api.operators.StreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>

Overrides:

open in class DoFnOperator<InputT,OutputT>

Throws:

java.lang.Exception

notifyCheckpointComplete

public final void notifyCheckpointComplete(long checkpointId)
                                    throws java.lang.Exception

Specified by:

notifyCheckpointComplete in interface org.apache.flink.runtime.state.CheckpointListener

Overrides:

notifyCheckpointComplete in class DoFnOperator<InputT,OutputT>

Throws:

java.lang.Exception

setKeyContextElement1

public void setKeyContextElement1(org.apache.flink.streaming.runtime.streamrecord.StreamRecord record)

Note: This is only relevant when we have a stateful DoFn. We want to control the key of the state backend ourselves and we must avoid any concurrent setting of the current active key. By overwriting this, we also prevent unnecessary serialization as the key has to be encoded as a byte array.

Specified by:

setKeyContextElement1 in interface org.apache.flink.streaming.api.operators.StreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>

Overrides:

setKeyContextElement1 in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>

setCurrentKey

public void setCurrentKey(java.lang.Object key)

We don't want to set anything here. This is due to asynchronous nature of processing elements from the SDK Harness. The Flink runtime sets the current key before calling processElement, but this does not work when sending elements to the SDK harness which may be processed at an arbitrary later point in time. State for keys is also accessed asynchronously via state requests.

We set the key only as it is required for 1) State requests 2) Timers (setting/firing).

Specified by:

setCurrentKey in interface org.apache.flink.streaming.api.operators.KeyContext

Overrides:

setCurrentKey in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>

getCurrentKey

public java.nio.ByteBuffer getCurrentKey()

Specified by:

getCurrentKey in interface org.apache.flink.streaming.api.operators.KeyContext

Overrides:

getCurrentKey in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>

fireTimerInternal

protected void fireTimerInternal(java.nio.ByteBuffer key,
                                 org.apache.beam.runners.core.TimerInternals.TimerData timer)

Overrides:

fireTimerInternal in class DoFnOperator<InputT,OutputT>

flushData

public void flushData()
               throws java.lang.Exception

Description copied from class: AbstractStreamOperatorCompat

Flush all remaining buffered data.

Throws:

java.lang.Exception

cleanUp

public void cleanUp()
             throws java.lang.Exception

Description copied from class: AbstractStreamOperatorCompat

Release all of the operator's resources.

Throws:

java.lang.Exception

addSideInputValue

protected void addSideInputValue(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.beam.sdk.transforms.join.RawUnionValue> streamRecord)

Description copied from class: DoFnOperator

Add the side input value. Here we are assuming that views have already been materialized and are sent over the wire as Iterable. Subclasses may elect to perform materialization in state and receive side input incrementally instead.

Overrides:

addSideInputValue in class DoFnOperator<InputT,OutputT>

createWrappingDoFnRunner

protected org.apache.beam.runners.core.DoFnRunner<InputT,OutputT> createWrappingDoFnRunner(org.apache.beam.runners.core.DoFnRunner<InputT,OutputT> wrappedRunner,
                                                                                           org.apache.beam.runners.core.StepContext stepContext)

Overrides:

createWrappingDoFnRunner in class DoFnOperator<InputT,OutputT>

applyInputWatermarkHold

public long applyInputWatermarkHold(long inputWatermark)

Description copied from class: DoFnOperator

Allows to apply a hold to the input watermark. By default, just passes the input watermark through.

Overrides:

applyInputWatermarkHold in class DoFnOperator<InputT,OutputT>

applyOutputWatermarkHold

public long applyOutputWatermarkHold(long currentOutputWatermark,
                                     long potentialOutputWatermark)

Description copied from class: DoFnOperator

Allows to apply a hold to the output watermark before it is send out. By default, just passes the potential output watermark through which will make it the new output watermark.

Overrides:

applyOutputWatermarkHold in class DoFnOperator<InputT,OutputT>

Parameters:

currentOutputWatermark - the current output watermark

potentialOutputWatermark - The potential new output watermark which can be adjusted, if needed. The input watermark hold has already been applied.

Returns:

The new output watermark which will be emitted.