InputT - the input type of the DoFnOutputT - the output type of the DoFnpublic class DoFnOperator<InputT,OutputT> extends AbstractStreamOperatorCompat<org.apache.beam.sdk.util.WindowedValue<OutputT>> implements org.apache.flink.streaming.api.operators.OneInputStreamOperator<org.apache.beam.sdk.util.WindowedValue<InputT>,org.apache.beam.sdk.util.WindowedValue<OutputT>>, org.apache.flink.streaming.api.operators.TwoInputStreamOperator<org.apache.beam.sdk.util.WindowedValue<InputT>,org.apache.beam.sdk.transforms.join.RawUnionValue,org.apache.beam.sdk.util.WindowedValue<OutputT>>, org.apache.flink.streaming.api.operators.Triggerable<java.nio.ByteBuffer,org.apache.beam.runners.core.TimerInternals.TimerData>
DoFns.| Modifier and Type | Class and Description |
|---|---|
static class |
DoFnOperator.BufferedOutputManager<OutputT>
A
DoFnRunners.OutputManager that can buffer its outputs. |
protected class |
DoFnOperator.FlinkStepContext
StepContext for running DoFns on Flink. |
static class |
DoFnOperator.MultiOutputOutputManagerFactory<OutputT>
Implementation of
OutputManagerFactory that creates an DoFnOperator.BufferedOutputManager
that can write to multiple logical outputs by Flink side output. |
| Modifier and Type | Field and Description |
|---|---|
protected java.util.List<org.apache.beam.sdk.values.TupleTag<?>> |
additionalOutputTags |
protected BufferingDoFnRunner<InputT,OutputT> |
bufferingDoFnRunner |
protected org.apache.beam.sdk.transforms.DoFn<InputT,OutputT> |
doFn |
protected org.apache.beam.runners.core.DoFnRunner<InputT,OutputT> |
doFnRunner |
protected org.apache.beam.sdk.coders.Coder<?> |
keyCoder |
protected FlinkStateInternals<?> |
keyedStateInternals |
protected org.apache.beam.sdk.values.TupleTag<OutputT> |
mainOutputTag |
protected DoFnOperator.BufferedOutputManager<OutputT> |
outputManager |
protected org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.OutputManagerFactory<OutputT> |
outputManagerFactory |
protected org.apache.beam.runners.core.PushbackSideInputDoFnRunner<InputT,OutputT> |
pushbackDoFnRunner |
protected org.apache.beam.runners.core.construction.SerializablePipelineOptions |
serializedOptions |
protected org.apache.beam.runners.core.SideInputHandler |
sideInputHandler |
protected org.apache.beam.runners.core.SideInputReader |
sideInputReader |
protected java.util.Collection<org.apache.beam.sdk.values.PCollectionView<?>> |
sideInputs |
protected java.util.Map<java.lang.Integer,org.apache.beam.sdk.values.PCollectionView<?>> |
sideInputTagMapping |
protected java.lang.String |
stepName |
protected org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.FlinkTimerInternals |
timerInternals |
protected org.apache.flink.streaming.api.operators.InternalTimerService<org.apache.beam.runners.core.TimerInternals.TimerData> |
timerService |
protected org.apache.beam.sdk.values.WindowingStrategy<?,?> |
windowingStrategy |
| Constructor and Description |
|---|
DoFnOperator(org.apache.beam.sdk.transforms.DoFn<InputT,OutputT> doFn,
java.lang.String stepName,
org.apache.beam.sdk.coders.Coder<org.apache.beam.sdk.util.WindowedValue<InputT>> inputWindowedCoder,
java.util.Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.coders.Coder<?>> outputCoders,
org.apache.beam.sdk.values.TupleTag<OutputT> mainOutputTag,
java.util.List<org.apache.beam.sdk.values.TupleTag<?>> additionalOutputTags,
org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.OutputManagerFactory<OutputT> outputManagerFactory,
org.apache.beam.sdk.values.WindowingStrategy<?,?> windowingStrategy,
java.util.Map<java.lang.Integer,org.apache.beam.sdk.values.PCollectionView<?>> sideInputTagMapping,
java.util.Collection<org.apache.beam.sdk.values.PCollectionView<?>> sideInputs,
org.apache.beam.sdk.options.PipelineOptions options,
org.apache.beam.sdk.coders.Coder<?> keyCoder,
org.apache.flink.api.java.functions.KeySelector<org.apache.beam.sdk.util.WindowedValue<InputT>,?> keySelector,
org.apache.beam.sdk.transforms.DoFnSchemaInformation doFnSchemaInformation,
java.util.Map<java.lang.String,org.apache.beam.sdk.values.PCollectionView<?>> sideInputMapping)
Constructor for DoFnOperator.
|
| Modifier and Type | Method and Description |
|---|---|
protected void |
addSideInputValue(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.beam.sdk.transforms.join.RawUnionValue> streamRecord)
Add the side input value.
|
long |
applyInputWatermarkHold(long inputWatermark)
Allows to apply a hold to the input watermark.
|
long |
applyOutputWatermarkHold(long currentOutputWatermark,
long potentialOutputWatermark)
Allows to apply a hold to the output watermark before it is send out.
|
protected org.apache.beam.runners.core.DoFnRunner<InputT,OutputT> |
createWrappingDoFnRunner(org.apache.beam.runners.core.DoFnRunner<InputT,OutputT> wrappedRunner,
org.apache.beam.runners.core.StepContext stepContext) |
protected void |
fireTimer(org.apache.beam.runners.core.TimerInternals.TimerData timerData) |
protected void |
fireTimerInternal(java.nio.ByteBuffer key,
org.apache.beam.runners.core.TimerInternals.TimerData timerData) |
org.apache.beam.sdk.transforms.DoFn.BundleFinalizer |
getBundleFinalizer() |
long |
getCurrentOutputWatermark() |
protected org.apache.beam.sdk.transforms.DoFn<InputT,OutputT> |
getDoFn() |
long |
getEffectiveInputWatermark() |
protected java.util.concurrent.locks.Lock |
getLockToAcquireForStateAccessDuringBundles()
Subclasses may provide a lock to ensure that the state backend is not accessed concurrently
during bundle execution.
|
void |
initializeState(org.apache.flink.runtime.state.StateInitializationContext context) |
protected void |
invokeFinishBundle() |
void |
notifyCheckpointComplete(long checkpointId) |
void |
onEventTime(org.apache.flink.streaming.api.operators.InternalTimer<java.nio.ByteBuffer,org.apache.beam.runners.core.TimerInternals.TimerData> timer) |
void |
onProcessingTime(org.apache.flink.streaming.api.operators.InternalTimer<java.nio.ByteBuffer,org.apache.beam.runners.core.TimerInternals.TimerData> timer) |
void |
open() |
void |
prepareSnapshotPreBarrier(long checkpointId) |
void |
processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.beam.sdk.util.WindowedValue<InputT>> streamRecord) |
void |
processElement1(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.beam.sdk.util.WindowedValue<InputT>> streamRecord) |
void |
processElement2(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.beam.sdk.transforms.join.RawUnionValue> streamRecord) |
void |
processWatermark(org.apache.flink.streaming.api.watermark.Watermark mark) |
void |
processWatermark1(org.apache.flink.streaming.api.watermark.Watermark mark) |
void |
processWatermark2(org.apache.flink.streaming.api.watermark.Watermark mark) |
protected void |
setBundleFinishedCallback(java.lang.Runnable callback) |
protected void |
setPreBundleCallback(java.lang.Runnable callback) |
void |
setup(org.apache.flink.streaming.runtime.tasks.StreamTask<?,?> containingTask,
org.apache.flink.streaming.api.graph.StreamConfig config,
org.apache.flink.streaming.api.operators.Output<org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.beam.sdk.util.WindowedValue<OutputT>>> output) |
void |
snapshotState(org.apache.flink.runtime.state.StateSnapshotContext context) |
close, dispose, getTimeServiceManagerCompatgetChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, numEventTimeTimers, numProcessingTimeTimers, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, reportOrForwardLatencyMarker, setChainingStrategy, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, snapshotStateclone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitprocessLatencyMarkerprocessLatencyMarker1, processLatencyMarker2close, dispose, getMetricGroup, getOperatorID, initializeState, setKeyContextElement1, setKeyContextElement2, snapshotStateprotected final org.apache.beam.runners.core.construction.SerializablePipelineOptions serializedOptions
protected final org.apache.beam.sdk.values.TupleTag<OutputT> mainOutputTag
protected final java.util.List<org.apache.beam.sdk.values.TupleTag<?>> additionalOutputTags
protected final java.util.Collection<org.apache.beam.sdk.values.PCollectionView<?>> sideInputs
protected final java.util.Map<java.lang.Integer,org.apache.beam.sdk.values.PCollectionView<?>> sideInputTagMapping
protected final org.apache.beam.sdk.values.WindowingStrategy<?,?> windowingStrategy
protected final org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.OutputManagerFactory<OutputT> outputManagerFactory
protected transient org.apache.beam.runners.core.PushbackSideInputDoFnRunner<InputT,OutputT> pushbackDoFnRunner
protected transient BufferingDoFnRunner<InputT,OutputT> bufferingDoFnRunner
protected transient org.apache.beam.runners.core.SideInputHandler sideInputHandler
protected transient org.apache.beam.runners.core.SideInputReader sideInputReader
protected transient DoFnOperator.BufferedOutputManager<OutputT> outputManager
protected transient FlinkStateInternals<?> keyedStateInternals
protected transient org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.FlinkTimerInternals timerInternals
protected final java.lang.String stepName
protected final org.apache.beam.sdk.coders.Coder<?> keyCoder
protected transient org.apache.flink.streaming.api.operators.InternalTimerService<org.apache.beam.runners.core.TimerInternals.TimerData> timerService
public DoFnOperator(org.apache.beam.sdk.transforms.DoFn<InputT,OutputT> doFn, java.lang.String stepName, org.apache.beam.sdk.coders.Coder<org.apache.beam.sdk.util.WindowedValue<InputT>> inputWindowedCoder, java.util.Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.coders.Coder<?>> outputCoders, org.apache.beam.sdk.values.TupleTag<OutputT> mainOutputTag, java.util.List<org.apache.beam.sdk.values.TupleTag<?>> additionalOutputTags, org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.OutputManagerFactory<OutputT> outputManagerFactory, org.apache.beam.sdk.values.WindowingStrategy<?,?> windowingStrategy, java.util.Map<java.lang.Integer,org.apache.beam.sdk.values.PCollectionView<?>> sideInputTagMapping, java.util.Collection<org.apache.beam.sdk.values.PCollectionView<?>> sideInputs, org.apache.beam.sdk.options.PipelineOptions options, org.apache.beam.sdk.coders.Coder<?> keyCoder, org.apache.flink.api.java.functions.KeySelector<org.apache.beam.sdk.util.WindowedValue<InputT>,?> keySelector, org.apache.beam.sdk.transforms.DoFnSchemaInformation doFnSchemaInformation, java.util.Map<java.lang.String,org.apache.beam.sdk.values.PCollectionView<?>> sideInputMapping)
protected org.apache.beam.runners.core.DoFnRunner<InputT,OutputT> createWrappingDoFnRunner(org.apache.beam.runners.core.DoFnRunner<InputT,OutputT> wrappedRunner, org.apache.beam.runners.core.StepContext stepContext)
public void setup(org.apache.flink.streaming.runtime.tasks.StreamTask<?,?> containingTask,
org.apache.flink.streaming.api.graph.StreamConfig config,
org.apache.flink.streaming.api.operators.Output<org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.beam.sdk.util.WindowedValue<OutputT>>> output)
public void initializeState(org.apache.flink.runtime.state.StateInitializationContext context)
throws java.lang.Exception
initializeState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperatorinitializeState in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>java.lang.Exceptionprotected java.util.concurrent.locks.Lock getLockToAcquireForStateAccessDuringBundles()
public void open()
throws java.lang.Exception
open in interface org.apache.flink.streaming.api.operators.StreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>open in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>java.lang.Exceptionpublic long getEffectiveInputWatermark()
public long getCurrentOutputWatermark()
protected final void setPreBundleCallback(java.lang.Runnable callback)
protected final void setBundleFinishedCallback(java.lang.Runnable callback)
public final void processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.beam.sdk.util.WindowedValue<InputT>> streamRecord)
public final void processElement1(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.beam.sdk.util.WindowedValue<InputT>> streamRecord) throws java.lang.Exception
protected void addSideInputValue(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.beam.sdk.transforms.join.RawUnionValue> streamRecord)
Iterable. Subclasses may elect to perform materialization in
state and receive side input incrementally instead.streamRecord - public final void processElement2(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.beam.sdk.transforms.join.RawUnionValue> streamRecord)
throws java.lang.Exception
public final void processWatermark(org.apache.flink.streaming.api.watermark.Watermark mark)
throws java.lang.Exception
processWatermark in interface org.apache.flink.streaming.api.operators.OneInputStreamOperator<org.apache.beam.sdk.util.WindowedValue<InputT>,org.apache.beam.sdk.util.WindowedValue<OutputT>>processWatermark in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>java.lang.Exceptionpublic final void processWatermark1(org.apache.flink.streaming.api.watermark.Watermark mark)
throws java.lang.Exception
processWatermark1 in interface org.apache.flink.streaming.api.operators.TwoInputStreamOperator<org.apache.beam.sdk.util.WindowedValue<InputT>,org.apache.beam.sdk.transforms.join.RawUnionValue,org.apache.beam.sdk.util.WindowedValue<OutputT>>processWatermark1 in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>java.lang.Exceptionpublic long applyInputWatermarkHold(long inputWatermark)
public long applyOutputWatermarkHold(long currentOutputWatermark,
long potentialOutputWatermark)
currentOutputWatermark - the current output watermarkpotentialOutputWatermark - The potential new output watermark which can be adjusted, if
needed. The input watermark hold has already been applied.public final void processWatermark2(org.apache.flink.streaming.api.watermark.Watermark mark)
throws java.lang.Exception
processWatermark2 in interface org.apache.flink.streaming.api.operators.TwoInputStreamOperator<org.apache.beam.sdk.util.WindowedValue<InputT>,org.apache.beam.sdk.transforms.join.RawUnionValue,org.apache.beam.sdk.util.WindowedValue<OutputT>>processWatermark2 in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>java.lang.Exceptionprotected final void invokeFinishBundle()
public void prepareSnapshotPreBarrier(long checkpointId)
prepareSnapshotPreBarrier in interface org.apache.flink.streaming.api.operators.StreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>prepareSnapshotPreBarrier in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>public final void snapshotState(org.apache.flink.runtime.state.StateSnapshotContext context)
throws java.lang.Exception
snapshotState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperatorsnapshotState in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>java.lang.Exceptionpublic org.apache.beam.sdk.transforms.DoFn.BundleFinalizer getBundleFinalizer()
public void notifyCheckpointComplete(long checkpointId)
throws java.lang.Exception
notifyCheckpointComplete in interface org.apache.flink.runtime.state.CheckpointListenernotifyCheckpointComplete in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>java.lang.Exceptionpublic void onEventTime(org.apache.flink.streaming.api.operators.InternalTimer<java.nio.ByteBuffer,org.apache.beam.runners.core.TimerInternals.TimerData> timer)
onEventTime in interface org.apache.flink.streaming.api.operators.Triggerable<java.nio.ByteBuffer,org.apache.beam.runners.core.TimerInternals.TimerData>public void onProcessingTime(org.apache.flink.streaming.api.operators.InternalTimer<java.nio.ByteBuffer,org.apache.beam.runners.core.TimerInternals.TimerData> timer)
onProcessingTime in interface org.apache.flink.streaming.api.operators.Triggerable<java.nio.ByteBuffer,org.apache.beam.runners.core.TimerInternals.TimerData>protected void fireTimerInternal(java.nio.ByteBuffer key,
org.apache.beam.runners.core.TimerInternals.TimerData timerData)
protected void fireTimer(org.apache.beam.runners.core.TimerInternals.TimerData timerData)