org.apache.beam.runners.flink.translation.wrappers.streaming

Class ExecutableStageDoFnOperator<InputT,OutputT>

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>

org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator<InputT,OutputT>

org.apache.beam.runners.flink.translation.wrappers.streaming.ExecutableStageDoFnOperator<InputT,OutputT>

All Implemented Interfaces:

java.io.Serializable, org.apache.flink.runtime.state.CheckpointListener, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.OneInputStreamOperator<org.apache.beam.sdk.util.WindowedValue<InputT>,org.apache.beam.sdk.util.WindowedValue<OutputT>>, org.apache.flink.streaming.api.operators.StreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>, org.apache.flink.streaming.api.operators.Triggerable<java.lang.Object,org.apache.beam.runners.core.TimerInternals.TimerData>, org.apache.flink.streaming.api.operators.TwoInputStreamOperator<org.apache.beam.sdk.util.WindowedValue<InputT>,org.apache.beam.sdk.transforms.join.RawUnionValue,org.apache.beam.sdk.util.WindowedValue<OutputT>>, org.apache.flink.util.Disposable

public class ExecutableStageDoFnOperator<InputT,OutputT>
extends DoFnOperator<InputT,OutputT>

This operator is the streaming equivalent of the FlinkExecutableStageFunction. It sends all received elements to the SDK harness and emits the received back elements to the downstream operators. It also takes care of handling side inputs and state.

TODO Integrate support for progress updates and metrics

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator

DoFnOperator.BufferedOutputManager<OutputT>, DoFnOperator.FlinkStepContext, DoFnOperator.MultiOutputOutputManagerFactory<OutputT>

Nested classes/interfaces inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

org.apache.flink.streaming.api.operators.AbstractStreamOperator.CountingOutput

Field Summary

Fields inherited from class org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator

additionalOutputTags, bufferingDoFnRunner, currentInputWatermark, currentOutputWatermark, currentSideInputWatermark, doFn, doFnRunner, keyCoder, keyedStateInternals, mainOutputTag, outputManager, outputManagerFactory, pushbackDoFnRunner, serializedOptions, sideInputHandler, sideInputReader, sideInputs, sideInputTagMapping, stepName, timerInternals, timerService, windowingStrategy

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

chainingStrategy, config, latencyStats, metrics, output, timeServiceManager

Constructor Summary

Constructors

Constructor and Description

ExecutableStageDoFnOperator(java.lang.String stepName, org.apache.beam.sdk.coders.Coder<org.apache.beam.sdk.util.WindowedValue<InputT>> windowedInputCoder, org.apache.beam.sdk.coders.Coder<InputT> inputCoder, java.util.Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.coders.Coder<?>> outputCoders, org.apache.beam.sdk.values.TupleTag<OutputT> mainOutputTag, java.util.List<org.apache.beam.sdk.values.TupleTag<?>> additionalOutputTags, org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.OutputManagerFactory<OutputT> outputManagerFactory, java.util.Map<java.lang.Integer,org.apache.beam.sdk.values.PCollectionView<?>> sideInputTagMapping, java.util.Collection<org.apache.beam.sdk.values.PCollectionView<?>> sideInputs, java.util.Map<org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload.SideInputId,org.apache.beam.sdk.values.PCollectionView<?>> sideInputIds, org.apache.beam.sdk.options.PipelineOptions options, org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload payload, org.apache.beam.runners.fnexecution.provisioning.JobInfo jobInfo, FlinkExecutableStageContext.Factory contextFactory, java.util.Map<java.lang.String,org.apache.beam.sdk.values.TupleTag<?>> outputMap, org.apache.beam.sdk.values.WindowingStrategy windowingStrategy, org.apache.beam.sdk.coders.Coder keyCoder, org.apache.flink.api.java.functions.KeySelector<org.apache.beam.sdk.util.WindowedValue<InputT>,?> keySelector) Constructor.

Method Summary

Modifier and Type	Method and Description
protected void	addSideInputValue(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.beam.sdk.transforms.join.RawUnionValue> streamRecord) Add the side input value.
protected org.apache.beam.runners.core.DoFnRunner<InputT,OutputT>	createWrappingDoFnRunner(org.apache.beam.runners.core.DoFnRunner<InputT,OutputT> wrappedRunner)
void	dispose()
void	fireTimer(org.apache.flink.streaming.api.operators.InternalTimer<?,org.apache.beam.runners.core.TimerInternals.TimerData> timer)
java.nio.ByteBuffer	getCurrentKey()
protected java.util.concurrent.locks.Lock	getLockToAcquireForStateAccessDuringBundles() Subclasses may provide a lock to ensure that the state backend is not accessed concurrently during bundle execution.
void	open()
void	processWatermark(org.apache.flink.streaming.api.watermark.Watermark mark)
void	setCurrentKey(java.lang.Object key) We don't want to set anything here.
void	setKeyContextElement1(org.apache.flink.streaming.runtime.streamrecord.StreamRecord record) Note: This is only relevant when we have a stateful DoFn.

Methods inherited from class org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator

close, getDoFn, getPushbackWatermarkHold, initializeState, invokeFinishBundle, notifyCheckpointComplete, onEventTime, onProcessingTime, processElement, processElement1, processElement2, processWatermark1, processWatermark2, setBundleFinishedCallback, setPushedBackWatermark, setup, snapshotState

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

getChainingStrategy, getContainingTask, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getUserCodeClassloader, initializeState, numEventTimeTimers, numProcessingTimeTimers, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, reportOrForwardLatencyMarker, setChainingStrategy, setKeyContextElement2, snapshotState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

processLatencyMarker

Methods inherited from interface org.apache.flink.streaming.api.operators.TwoInputStreamOperator

processLatencyMarker1, processLatencyMarker2

Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

getChainingStrategy, getMetricGroup, getOperatorID, initializeState, setChainingStrategy, setKeyContextElement2, snapshotState

Constructor Detail

ExecutableStageDoFnOperator

public ExecutableStageDoFnOperator(java.lang.String stepName,
                                   org.apache.beam.sdk.coders.Coder<org.apache.beam.sdk.util.WindowedValue<InputT>> windowedInputCoder,
                                   org.apache.beam.sdk.coders.Coder<InputT> inputCoder,
                                   java.util.Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.coders.Coder<?>> outputCoders,
                                   org.apache.beam.sdk.values.TupleTag<OutputT> mainOutputTag,
                                   java.util.List<org.apache.beam.sdk.values.TupleTag<?>> additionalOutputTags,
                                   org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.OutputManagerFactory<OutputT> outputManagerFactory,
                                   java.util.Map<java.lang.Integer,org.apache.beam.sdk.values.PCollectionView<?>> sideInputTagMapping,
                                   java.util.Collection<org.apache.beam.sdk.values.PCollectionView<?>> sideInputs,
                                   java.util.Map<org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload.SideInputId,org.apache.beam.sdk.values.PCollectionView<?>> sideInputIds,
                                   org.apache.beam.sdk.options.PipelineOptions options,
                                   org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload payload,
                                   org.apache.beam.runners.fnexecution.provisioning.JobInfo jobInfo,
                                   FlinkExecutableStageContext.Factory contextFactory,
                                   java.util.Map<java.lang.String,org.apache.beam.sdk.values.TupleTag<?>> outputMap,
                                   org.apache.beam.sdk.values.WindowingStrategy windowingStrategy,
                                   org.apache.beam.sdk.coders.Coder keyCoder,
                                   org.apache.flink.api.java.functions.KeySelector<org.apache.beam.sdk.util.WindowedValue<InputT>,?> keySelector)

Constructor.

Method Detail

getLockToAcquireForStateAccessDuringBundles

protected java.util.concurrent.locks.Lock getLockToAcquireForStateAccessDuringBundles()

Description copied from class: DoFnOperator

Subclasses may provide a lock to ensure that the state backend is not accessed concurrently during bundle execution.

Overrides:

getLockToAcquireForStateAccessDuringBundles in class DoFnOperator<InputT,OutputT>

open

public void open()
          throws java.lang.Exception

Specified by:

open in interface org.apache.flink.streaming.api.operators.StreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>

Overrides:

open in class DoFnOperator<InputT,OutputT>

Throws:

java.lang.Exception

setKeyContextElement1

public void setKeyContextElement1(org.apache.flink.streaming.runtime.streamrecord.StreamRecord record)

Note: This is only relevant when we have a stateful DoFn. We want to control the key of the state backend ourselves and we must avoid any concurrent setting of the current active key. By overwriting this, we also prevent unnecessary serialization as the key has to be encoded as a byte array.

Specified by:

setKeyContextElement1 in interface org.apache.flink.streaming.api.operators.StreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>

Overrides:

setKeyContextElement1 in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>

setCurrentKey

public void setCurrentKey(java.lang.Object key)

We don't want to set anything here. This is due to asynchronous nature of processing elements from the SDK Harness. The Flink runtime sets the current key before calling processElement, but this does not work when sending elements to the SDK harness which may be processed at an arbitrary later point in time. State for keys is also accessed asynchronously via state requests.

We set the key only as it is required for 1) State requests 2) Timers (setting/firing).

Specified by:

setCurrentKey in interface org.apache.flink.streaming.api.operators.KeyContext

Overrides:

setCurrentKey in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>

getCurrentKey

public java.nio.ByteBuffer getCurrentKey()

Specified by:

getCurrentKey in interface org.apache.flink.streaming.api.operators.KeyContext

Overrides:

getCurrentKey in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>

fireTimer

public void fireTimer(org.apache.flink.streaming.api.operators.InternalTimer<?,org.apache.beam.runners.core.TimerInternals.TimerData> timer)

Overrides:

fireTimer in class DoFnOperator<InputT,OutputT>

dispose

public void dispose()
             throws java.lang.Exception

Specified by:

dispose in interface org.apache.flink.streaming.api.operators.StreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>

Specified by:

dispose in interface org.apache.flink.util.Disposable

Overrides:

dispose in class DoFnOperator<InputT,OutputT>

Throws:

java.lang.Exception

addSideInputValue

protected void addSideInputValue(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.beam.sdk.transforms.join.RawUnionValue> streamRecord)

Description copied from class: DoFnOperator

Add the side input value. Here we are assuming that views have already been materialized and are sent over the wire as Iterable. Subclasses may elect to perform materialization in state and receive side input incrementally instead.

Overrides:

addSideInputValue in class DoFnOperator<InputT,OutputT>

createWrappingDoFnRunner

protected org.apache.beam.runners.core.DoFnRunner<InputT,OutputT> createWrappingDoFnRunner(org.apache.beam.runners.core.DoFnRunner<InputT,OutputT> wrappedRunner)

Overrides:

createWrappingDoFnRunner in class DoFnOperator<InputT,OutputT>

processWatermark

public void processWatermark(org.apache.flink.streaming.api.watermark.Watermark mark)
                      throws java.lang.Exception

Specified by:

processWatermark in interface org.apache.flink.streaming.api.operators.OneInputStreamOperator<org.apache.beam.sdk.util.WindowedValue<InputT>,org.apache.beam.sdk.util.WindowedValue<OutputT>>

Overrides:

processWatermark in class DoFnOperator<InputT,OutputT>

Throws:

java.lang.Exception