org.apache.beam.runners.flink.translation.wrappers.streaming

Class DoFnOperator<InputT,OutputT>

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>

org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator<InputT,OutputT>

Type Parameters:

InputT - the input type of the DoFn

OutputT - the output type of the DoFn

All Implemented Interfaces:

java.io.Serializable, org.apache.flink.runtime.state.CheckpointListener, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.OneInputStreamOperator<org.apache.beam.sdk.util.WindowedValue<InputT>,org.apache.beam.sdk.util.WindowedValue<OutputT>>, org.apache.flink.streaming.api.operators.StreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>, org.apache.flink.streaming.api.operators.Triggerable<java.lang.Object,org.apache.beam.runners.core.TimerInternals.TimerData>, org.apache.flink.streaming.api.operators.TwoInputStreamOperator<org.apache.beam.sdk.util.WindowedValue<InputT>,org.apache.beam.sdk.transforms.join.RawUnionValue,org.apache.beam.sdk.util.WindowedValue<OutputT>>, org.apache.flink.util.Disposable

Direct Known Subclasses:

ExecutableStageDoFnOperator, SplittableDoFnOperator, WindowDoFnOperator

public class DoFnOperator<InputT,OutputT>
extends org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>
implements org.apache.flink.streaming.api.operators.OneInputStreamOperator<org.apache.beam.sdk.util.WindowedValue<InputT>,org.apache.beam.sdk.util.WindowedValue<OutputT>>, org.apache.flink.streaming.api.operators.TwoInputStreamOperator<org.apache.beam.sdk.util.WindowedValue<InputT>,org.apache.beam.sdk.transforms.join.RawUnionValue,org.apache.beam.sdk.util.WindowedValue<OutputT>>, org.apache.flink.streaming.api.operators.Triggerable<java.lang.Object,org.apache.beam.runners.core.TimerInternals.TimerData>

Flink operator for executing DoFns.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DoFnOperator.BufferedOutputManager<OutputT> A DoFnRunners.OutputManager that can buffer its outputs.
protected class	DoFnOperator.FlinkStepContext StepContext for running DoFns on Flink.
static class	DoFnOperator.MultiOutputOutputManagerFactory<OutputT> Implementation of OutputManagerFactory that creates an DoFnOperator.BufferedOutputManager that can write to multiple logical outputs by Flink side output.

Nested classes/interfaces inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

org.apache.flink.streaming.api.operators.AbstractStreamOperator.CountingOutput

Field Summary

Fields

Modifier and Type	Field and Description
protected java.util.List<org.apache.beam.sdk.values.TupleTag<?>>	additionalOutputTags
protected BufferingDoFnRunner<InputT,OutputT>	bufferingDoFnRunner
protected long	currentInputWatermark
protected long	currentOutputWatermark
protected long	currentSideInputWatermark
protected org.apache.beam.sdk.transforms.DoFn<InputT,OutputT>	doFn
protected org.apache.beam.runners.core.DoFnRunner<InputT,OutputT>	doFnRunner
protected org.apache.beam.sdk.coders.Coder<?>	keyCoder
protected FlinkStateInternals<?>	keyedStateInternals
protected org.apache.beam.sdk.values.TupleTag<OutputT>	mainOutputTag
protected DoFnOperator.BufferedOutputManager<OutputT>	outputManager
protected org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.OutputManagerFactory<OutputT>	outputManagerFactory
protected org.apache.beam.runners.core.PushbackSideInputDoFnRunner<InputT,OutputT>	pushbackDoFnRunner
protected org.apache.beam.runners.core.construction.SerializablePipelineOptions	serializedOptions
protected org.apache.beam.runners.core.SideInputHandler	sideInputHandler
protected org.apache.beam.runners.core.SideInputReader	sideInputReader
protected java.util.Collection<org.apache.beam.sdk.values.PCollectionView<?>>	sideInputs
protected java.util.Map<java.lang.Integer,org.apache.beam.sdk.values.PCollectionView<?>>	sideInputTagMapping
protected java.lang.String	stepName
protected org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.FlinkTimerInternals	timerInternals
protected org.apache.flink.streaming.api.operators.InternalTimerService<org.apache.beam.runners.core.TimerInternals.TimerData>	timerService
protected org.apache.beam.sdk.values.WindowingStrategy<?,?>	windowingStrategy

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

chainingStrategy, config, latencyStats, LOG, metrics, output, timeServiceManager

Constructor Summary

Constructors

Constructor and Description

DoFnOperator(org.apache.beam.sdk.transforms.DoFn<InputT,OutputT> doFn, java.lang.String stepName, org.apache.beam.sdk.coders.Coder<org.apache.beam.sdk.util.WindowedValue<InputT>> inputWindowedCoder, org.apache.beam.sdk.coders.Coder<InputT> inputCoder, java.util.Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.coders.Coder<?>> outputCoders, org.apache.beam.sdk.values.TupleTag<OutputT> mainOutputTag, java.util.List<org.apache.beam.sdk.values.TupleTag<?>> additionalOutputTags, org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.OutputManagerFactory<OutputT> outputManagerFactory, org.apache.beam.sdk.values.WindowingStrategy<?,?> windowingStrategy, java.util.Map<java.lang.Integer,org.apache.beam.sdk.values.PCollectionView<?>> sideInputTagMapping, java.util.Collection<org.apache.beam.sdk.values.PCollectionView<?>> sideInputs, org.apache.beam.sdk.options.PipelineOptions options, org.apache.beam.sdk.coders.Coder<?> keyCoder, org.apache.flink.api.java.functions.KeySelector<org.apache.beam.sdk.util.WindowedValue<InputT>,?> keySelector, org.apache.beam.sdk.transforms.DoFnSchemaInformation doFnSchemaInformation) Constructor for DoFnOperator.

Method Summary

Modifier and Type	Method and Description
protected void	addSideInputValue(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.beam.sdk.transforms.join.RawUnionValue> streamRecord) Add the side input value.
void	close()
protected org.apache.beam.runners.core.DoFnRunner<InputT,OutputT>	createWrappingDoFnRunner(org.apache.beam.runners.core.DoFnRunner<InputT,OutputT> wrappedRunner)
void	dispose()
void	fireTimer(org.apache.flink.streaming.api.operators.InternalTimer<?,org.apache.beam.runners.core.TimerInternals.TimerData> timer)
protected org.apache.beam.sdk.transforms.DoFn<InputT,OutputT>	getDoFn()
protected java.util.concurrent.locks.Lock	getLockToAcquireForStateAccessDuringBundles() Subclasses may provide a lock to ensure that the state backend is not accessed concurrently during bundle execution.
protected long	getPushbackWatermarkHold()
void	initializeState(org.apache.flink.runtime.state.StateInitializationContext context)
protected void	invokeFinishBundle()
void	notifyCheckpointComplete(long checkpointId)
void	onEventTime(org.apache.flink.streaming.api.operators.InternalTimer<java.lang.Object,org.apache.beam.runners.core.TimerInternals.TimerData> timer)
void	onProcessingTime(org.apache.flink.streaming.api.operators.InternalTimer<java.lang.Object,org.apache.beam.runners.core.TimerInternals.TimerData> timer)
void	open()
void	processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.beam.sdk.util.WindowedValue<InputT>> streamRecord)
void	processElement1(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.beam.sdk.util.WindowedValue<InputT>> streamRecord)
void	processElement2(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.beam.sdk.transforms.join.RawUnionValue> streamRecord)
void	processWatermark(org.apache.flink.streaming.api.watermark.Watermark mark)
void	processWatermark1(org.apache.flink.streaming.api.watermark.Watermark mark)
void	processWatermark2(org.apache.flink.streaming.api.watermark.Watermark mark)
protected void	setBundleFinishedCallback(java.lang.Runnable callback)
protected void	setPushedBackWatermark(long watermark)
void	setup(org.apache.flink.streaming.runtime.tasks.StreamTask<?,?> containingTask, org.apache.flink.streaming.api.graph.StreamConfig config, org.apache.flink.streaming.api.operators.Output<org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.beam.sdk.util.WindowedValue<OutputT>>> output)
void	snapshotState(org.apache.flink.runtime.state.StateSnapshotContext context)

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getUserCodeClassloader, initializeState, numEventTimeTimers, numProcessingTimeTimers, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, reportOrForwardLatencyMarker, setChainingStrategy, setCurrentKey, setKeyContextElement1, setKeyContextElement2, snapshotState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

processLatencyMarker

Methods inherited from interface org.apache.flink.streaming.api.operators.TwoInputStreamOperator

processLatencyMarker1, processLatencyMarker2

Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

getChainingStrategy, getMetricGroup, getOperatorID, initializeState, setChainingStrategy, setKeyContextElement1, setKeyContextElement2, snapshotState

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

getCurrentKey, setCurrentKey

Field Detail

doFn

protected org.apache.beam.sdk.transforms.DoFn<InputT,OutputT> doFn

serializedOptions

protected final org.apache.beam.runners.core.construction.SerializablePipelineOptions serializedOptions

mainOutputTag

protected final org.apache.beam.sdk.values.TupleTag<OutputT> mainOutputTag

additionalOutputTags

protected final java.util.List<org.apache.beam.sdk.values.TupleTag<?>> additionalOutputTags

sideInputs

protected final java.util.Collection<org.apache.beam.sdk.values.PCollectionView<?>> sideInputs

sideInputTagMapping

protected final java.util.Map<java.lang.Integer,org.apache.beam.sdk.values.PCollectionView<?>> sideInputTagMapping

windowingStrategy

protected final org.apache.beam.sdk.values.WindowingStrategy<?,?> windowingStrategy

outputManagerFactory

protected final org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.OutputManagerFactory<OutputT> outputManagerFactory

doFnRunner

protected transient org.apache.beam.runners.core.DoFnRunner<InputT,OutputT> doFnRunner

pushbackDoFnRunner

protected transient org.apache.beam.runners.core.PushbackSideInputDoFnRunner<InputT,OutputT> pushbackDoFnRunner

bufferingDoFnRunner

protected transient BufferingDoFnRunner<InputT,OutputT> bufferingDoFnRunner

sideInputHandler

protected transient org.apache.beam.runners.core.SideInputHandler sideInputHandler

sideInputReader

protected transient org.apache.beam.runners.core.SideInputReader sideInputReader

outputManager

protected transient DoFnOperator.BufferedOutputManager<OutputT> outputManager

currentInputWatermark

protected transient long currentInputWatermark

currentSideInputWatermark

protected transient long currentSideInputWatermark

currentOutputWatermark

protected transient long currentOutputWatermark

keyedStateInternals

protected transient FlinkStateInternals<?> keyedStateInternals

stepName

protected final java.lang.String stepName

keyCoder

protected final org.apache.beam.sdk.coders.Coder<?> keyCoder

timerService

protected transient org.apache.flink.streaming.api.operators.InternalTimerService<org.apache.beam.runners.core.TimerInternals.TimerData> timerService

timerInternals

protected transient org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.FlinkTimerInternals timerInternals

Constructor Detail

DoFnOperator

public DoFnOperator(org.apache.beam.sdk.transforms.DoFn<InputT,OutputT> doFn,
                    java.lang.String stepName,
                    org.apache.beam.sdk.coders.Coder<org.apache.beam.sdk.util.WindowedValue<InputT>> inputWindowedCoder,
                    org.apache.beam.sdk.coders.Coder<InputT> inputCoder,
                    java.util.Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.coders.Coder<?>> outputCoders,
                    org.apache.beam.sdk.values.TupleTag<OutputT> mainOutputTag,
                    java.util.List<org.apache.beam.sdk.values.TupleTag<?>> additionalOutputTags,
                    org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.OutputManagerFactory<OutputT> outputManagerFactory,
                    org.apache.beam.sdk.values.WindowingStrategy<?,?> windowingStrategy,
                    java.util.Map<java.lang.Integer,org.apache.beam.sdk.values.PCollectionView<?>> sideInputTagMapping,
                    java.util.Collection<org.apache.beam.sdk.values.PCollectionView<?>> sideInputs,
                    org.apache.beam.sdk.options.PipelineOptions options,
                    org.apache.beam.sdk.coders.Coder<?> keyCoder,
                    org.apache.flink.api.java.functions.KeySelector<org.apache.beam.sdk.util.WindowedValue<InputT>,?> keySelector,
                    org.apache.beam.sdk.transforms.DoFnSchemaInformation doFnSchemaInformation)

Constructor for DoFnOperator.

Method Detail

getDoFn

protected org.apache.beam.sdk.transforms.DoFn<InputT,OutputT> getDoFn()

createWrappingDoFnRunner

protected org.apache.beam.runners.core.DoFnRunner<InputT,OutputT> createWrappingDoFnRunner(org.apache.beam.runners.core.DoFnRunner<InputT,OutputT> wrappedRunner)

setup

public void setup(org.apache.flink.streaming.runtime.tasks.StreamTask<?,?> containingTask,
                  org.apache.flink.streaming.api.graph.StreamConfig config,
                  org.apache.flink.streaming.api.operators.Output<org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.beam.sdk.util.WindowedValue<OutputT>>> output)

Specified by:

setup in interface org.apache.flink.streaming.api.operators.StreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>

Overrides:

setup in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>

initializeState

public void initializeState(org.apache.flink.runtime.state.StateInitializationContext context)
                     throws java.lang.Exception

Overrides:

initializeState in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>

Throws:

java.lang.Exception

getLockToAcquireForStateAccessDuringBundles

protected java.util.concurrent.locks.Lock getLockToAcquireForStateAccessDuringBundles()

Subclasses may provide a lock to ensure that the state backend is not accessed concurrently during bundle execution.

open

public void open()
          throws java.lang.Exception

Specified by:

open in interface org.apache.flink.streaming.api.operators.StreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>

Overrides:

open in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>

Throws:

java.lang.Exception

dispose

public void dispose()
             throws java.lang.Exception

Specified by:

dispose in interface org.apache.flink.streaming.api.operators.StreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>

Specified by:

dispose in interface org.apache.flink.util.Disposable

Overrides:

dispose in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>

Throws:

java.lang.Exception

close

public void close()
           throws java.lang.Exception

Specified by:

close in interface org.apache.flink.streaming.api.operators.StreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>

Overrides:

close in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>

Throws:

java.lang.Exception

getPushbackWatermarkHold

protected long getPushbackWatermarkHold()

setPushedBackWatermark

protected void setPushedBackWatermark(long watermark)

setBundleFinishedCallback

protected void setBundleFinishedCallback(java.lang.Runnable callback)

processElement

public final void processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.beam.sdk.util.WindowedValue<InputT>> streamRecord)
                          throws java.lang.Exception

Specified by:

processElement in interface org.apache.flink.streaming.api.operators.OneInputStreamOperator<org.apache.beam.sdk.util.WindowedValue<InputT>,org.apache.beam.sdk.util.WindowedValue<OutputT>>

Throws:

java.lang.Exception

processElement1

public final void processElement1(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.beam.sdk.util.WindowedValue<InputT>> streamRecord)
                           throws java.lang.Exception

Specified by:

processElement1 in interface org.apache.flink.streaming.api.operators.TwoInputStreamOperator<org.apache.beam.sdk.util.WindowedValue<InputT>,org.apache.beam.sdk.transforms.join.RawUnionValue,org.apache.beam.sdk.util.WindowedValue<OutputT>>

Throws:

java.lang.Exception

addSideInputValue

protected void addSideInputValue(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.beam.sdk.transforms.join.RawUnionValue> streamRecord)

Add the side input value. Here we are assuming that views have already been materialized and are sent over the wire as Iterable. Subclasses may elect to perform materialization in state and receive side input incrementally instead.

Parameters:

streamRecord -

processElement2

public final void processElement2(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.beam.sdk.transforms.join.RawUnionValue> streamRecord)
                           throws java.lang.Exception

Specified by:

processElement2 in interface org.apache.flink.streaming.api.operators.TwoInputStreamOperator<org.apache.beam.sdk.util.WindowedValue<InputT>,org.apache.beam.sdk.transforms.join.RawUnionValue,org.apache.beam.sdk.util.WindowedValue<OutputT>>

Throws:

java.lang.Exception

processWatermark

public void processWatermark(org.apache.flink.streaming.api.watermark.Watermark mark)
                      throws java.lang.Exception

Specified by:

processWatermark in interface org.apache.flink.streaming.api.operators.OneInputStreamOperator<org.apache.beam.sdk.util.WindowedValue<InputT>,org.apache.beam.sdk.util.WindowedValue<OutputT>>

Overrides:

processWatermark in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>

Throws:

java.lang.Exception

processWatermark1

public void processWatermark1(org.apache.flink.streaming.api.watermark.Watermark mark)
                       throws java.lang.Exception

Specified by:

processWatermark1 in interface org.apache.flink.streaming.api.operators.TwoInputStreamOperator<org.apache.beam.sdk.util.WindowedValue<InputT>,org.apache.beam.sdk.transforms.join.RawUnionValue,org.apache.beam.sdk.util.WindowedValue<OutputT>>

Overrides:

processWatermark1 in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>

Throws:

java.lang.Exception

processWatermark2

public void processWatermark2(org.apache.flink.streaming.api.watermark.Watermark mark)
                       throws java.lang.Exception

Specified by:

processWatermark2 in interface org.apache.flink.streaming.api.operators.TwoInputStreamOperator<org.apache.beam.sdk.util.WindowedValue<InputT>,org.apache.beam.sdk.transforms.join.RawUnionValue,org.apache.beam.sdk.util.WindowedValue<OutputT>>

Overrides:

processWatermark2 in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>

Throws:

java.lang.Exception

invokeFinishBundle

protected final void invokeFinishBundle()

snapshotState

public final void snapshotState(org.apache.flink.runtime.state.StateSnapshotContext context)
                         throws java.lang.Exception

Overrides:

snapshotState in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>

Throws:

java.lang.Exception

notifyCheckpointComplete

public final void notifyCheckpointComplete(long checkpointId)
                                    throws java.lang.Exception

Specified by:

notifyCheckpointComplete in interface org.apache.flink.runtime.state.CheckpointListener

Overrides:

notifyCheckpointComplete in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>

Throws:

java.lang.Exception

onEventTime

public void onEventTime(org.apache.flink.streaming.api.operators.InternalTimer<java.lang.Object,org.apache.beam.runners.core.TimerInternals.TimerData> timer)
                 throws java.lang.Exception

Specified by:

onEventTime in interface org.apache.flink.streaming.api.operators.Triggerable<java.lang.Object,org.apache.beam.runners.core.TimerInternals.TimerData>

Throws:

java.lang.Exception

onProcessingTime

public void onProcessingTime(org.apache.flink.streaming.api.operators.InternalTimer<java.lang.Object,org.apache.beam.runners.core.TimerInternals.TimerData> timer)
                      throws java.lang.Exception

Specified by:

onProcessingTime in interface org.apache.flink.streaming.api.operators.Triggerable<java.lang.Object,org.apache.beam.runners.core.TimerInternals.TimerData>

Throws:

java.lang.Exception

fireTimer

public void fireTimer(org.apache.flink.streaming.api.operators.InternalTimer<?,org.apache.beam.runners.core.TimerInternals.TimerData> timer)