org.apache.beam.runners.flink.translation.wrappers.streaming

Class SplittableDoFnOperator<InputT,OutputT,RestrictionT>

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<OutputT>

org.apache.beam.runners.flink.translation.wrappers.streaming.AbstractStreamOperatorCompat<org.apache.beam.sdk.util.WindowedValue<OutputT>>

org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator<org.apache.beam.runners.core.KeyedWorkItem<byte[],org.apache.beam.sdk.values.KV<InputT,RestrictionT>>,OutputT>

org.apache.beam.runners.flink.translation.wrappers.streaming.SplittableDoFnOperator<InputT,OutputT,RestrictionT>

All Implemented Interfaces:

java.io.Serializable, org.apache.flink.runtime.state.CheckpointListener, org.apache.flink.streaming.api.operators.KeyContext, org.apache.flink.streaming.api.operators.OneInputStreamOperator<org.apache.beam.sdk.util.WindowedValue<org.apache.beam.runners.core.KeyedWorkItem<byte[],org.apache.beam.sdk.values.KV<InputT,RestrictionT>>>,org.apache.beam.sdk.util.WindowedValue<OutputT>>, org.apache.flink.streaming.api.operators.SetupableStreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>, org.apache.flink.streaming.api.operators.StreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>, org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator, org.apache.flink.streaming.api.operators.Triggerable<java.nio.ByteBuffer,org.apache.beam.runners.core.TimerInternals.TimerData>, org.apache.flink.streaming.api.operators.TwoInputStreamOperator<org.apache.beam.sdk.util.WindowedValue<org.apache.beam.runners.core.KeyedWorkItem<byte[],org.apache.beam.sdk.values.KV<InputT,RestrictionT>>>,org.apache.beam.sdk.transforms.join.RawUnionValue,org.apache.beam.sdk.util.WindowedValue<OutputT>>, org.apache.flink.util.Disposable

public class SplittableDoFnOperator<InputT,OutputT,RestrictionT>
extends DoFnOperator<org.apache.beam.runners.core.KeyedWorkItem<byte[],org.apache.beam.sdk.values.KV<InputT,RestrictionT>>,OutputT>

Flink operator for executing splittable DoFns. Specifically, for executing the @ProcessElement method of a splittable DoFn.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator

DoFnOperator.BufferedOutputManager<OutputT>, DoFnOperator.FlinkStepContext, DoFnOperator.MultiOutputOutputManagerFactory<OutputT>

Field Summary

Fields inherited from class org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator

additionalOutputTags, bufferingDoFnRunner, doFn, doFnRunner, keyCoder, keyedStateInternals, mainOutputTag, outputManager, outputManagerFactory, pushbackDoFnRunner, serializedOptions, sideInputHandler, sideInputReader, sideInputs, sideInputTagMapping, stepName, timerInternals, timerService, windowingStrategy

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

chainingStrategy, config, latencyStats, metrics, output, processingTimeService

Constructor Summary

Constructors

Constructor and Description

SplittableDoFnOperator(org.apache.beam.sdk.transforms.DoFn<org.apache.beam.runners.core.KeyedWorkItem<byte[],org.apache.beam.sdk.values.KV<InputT,RestrictionT>>,OutputT> doFn, java.lang.String stepName, org.apache.beam.sdk.coders.Coder<org.apache.beam.sdk.util.WindowedValue<org.apache.beam.runners.core.KeyedWorkItem<byte[],org.apache.beam.sdk.values.KV<InputT,RestrictionT>>>> windowedInputCoder, java.util.Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.coders.Coder<?>> outputCoders, org.apache.beam.sdk.values.TupleTag<OutputT> mainOutputTag, java.util.List<org.apache.beam.sdk.values.TupleTag<?>> additionalOutputTags, org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.OutputManagerFactory<OutputT> outputManagerFactory, org.apache.beam.sdk.values.WindowingStrategy<?,?> windowingStrategy, java.util.Map<java.lang.Integer,org.apache.beam.sdk.values.PCollectionView<?>> sideInputTagMapping, java.util.Collection<org.apache.beam.sdk.values.PCollectionView<?>> sideInputs, org.apache.beam.sdk.options.PipelineOptions options, org.apache.beam.sdk.coders.Coder<?> keyCoder, org.apache.flink.api.java.functions.KeySelector<org.apache.beam.sdk.util.WindowedValue<org.apache.beam.runners.core.KeyedWorkItem<byte[],org.apache.beam.sdk.values.KV<InputT,RestrictionT>>>,?> keySelector)

Method Summary

Modifier and Type	Method and Description
void	close()
protected org.apache.beam.runners.core.DoFnRunner<org.apache.beam.runners.core.KeyedWorkItem<byte[],org.apache.beam.sdk.values.KV<InputT,RestrictionT>>,OutputT>	createWrappingDoFnRunner(org.apache.beam.runners.core.DoFnRunner<org.apache.beam.runners.core.KeyedWorkItem<byte[],org.apache.beam.sdk.values.KV<InputT,RestrictionT>>,OutputT> wrappedRunner, org.apache.beam.runners.core.StepContext stepContext)
protected void	fireTimer(org.apache.beam.runners.core.TimerInternals.TimerData timer)
void	initializeState(org.apache.flink.runtime.state.StateInitializationContext context)

Methods inherited from class org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator

addSideInputValue, applyInputWatermarkHold, applyOutputWatermarkHold, fireTimerInternal, getBundleFinalizer, getCurrentOutputWatermark, getDoFn, getEffectiveInputWatermark, getLockToAcquireForStateAccessDuringBundles, invokeFinishBundle, notifyCheckpointComplete, onEventTime, onProcessingTime, open, prepareSnapshotPreBarrier, processElement, processElement1, processElement2, processWatermark, processWatermark1, processWatermark2, setBundleFinishedCallback, setPreBundleCallback, setup, snapshotState

Methods inherited from class org.apache.beam.runners.flink.translation.wrappers.streaming.AbstractStreamOperatorCompat

dispose, getTimeServiceManagerCompat

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

getChainingStrategy, getContainingTask, getCurrentKey, getExecutionConfig, getInternalTimerService, getKeyedStateBackend, getKeyedStateStore, getMetricGroup, getOperatorConfig, getOperatorID, getOperatorName, getOperatorStateBackend, getOrCreateKeyedState, getPartitionedState, getPartitionedState, getProcessingTimeService, getRuntimeContext, getTimeServiceManager, getUserCodeClassloader, initializeState, isUsingCustomRawKeyedState, notifyCheckpointAborted, numEventTimeTimers, numProcessingTimeTimers, processLatencyMarker, processLatencyMarker1, processLatencyMarker2, reportOrForwardLatencyMarker, setChainingStrategy, setCurrentKey, setKeyContextElement1, setKeyContextElement2, setProcessingTimeService, snapshotState

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.operators.OneInputStreamOperator

processLatencyMarker

Methods inherited from interface org.apache.flink.streaming.api.operators.TwoInputStreamOperator

processLatencyMarker1, processLatencyMarker2

Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

dispose, getMetricGroup, getOperatorID, initializeState, setKeyContextElement1, setKeyContextElement2, snapshotState

Methods inherited from interface org.apache.flink.runtime.state.CheckpointListener

notifyCheckpointAborted

Methods inherited from interface org.apache.flink.streaming.api.operators.KeyContext

getCurrentKey, setCurrentKey

Constructor Detail

SplittableDoFnOperator

public SplittableDoFnOperator(org.apache.beam.sdk.transforms.DoFn<org.apache.beam.runners.core.KeyedWorkItem<byte[],org.apache.beam.sdk.values.KV<InputT,RestrictionT>>,OutputT> doFn,
                              java.lang.String stepName,
                              org.apache.beam.sdk.coders.Coder<org.apache.beam.sdk.util.WindowedValue<org.apache.beam.runners.core.KeyedWorkItem<byte[],org.apache.beam.sdk.values.KV<InputT,RestrictionT>>>> windowedInputCoder,
                              java.util.Map<org.apache.beam.sdk.values.TupleTag<?>,org.apache.beam.sdk.coders.Coder<?>> outputCoders,
                              org.apache.beam.sdk.values.TupleTag<OutputT> mainOutputTag,
                              java.util.List<org.apache.beam.sdk.values.TupleTag<?>> additionalOutputTags,
                              org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.OutputManagerFactory<OutputT> outputManagerFactory,
                              org.apache.beam.sdk.values.WindowingStrategy<?,?> windowingStrategy,
                              java.util.Map<java.lang.Integer,org.apache.beam.sdk.values.PCollectionView<?>> sideInputTagMapping,
                              java.util.Collection<org.apache.beam.sdk.values.PCollectionView<?>> sideInputs,
                              org.apache.beam.sdk.options.PipelineOptions options,
                              org.apache.beam.sdk.coders.Coder<?> keyCoder,
                              org.apache.flink.api.java.functions.KeySelector<org.apache.beam.sdk.util.WindowedValue<org.apache.beam.runners.core.KeyedWorkItem<byte[],org.apache.beam.sdk.values.KV<InputT,RestrictionT>>>,?> keySelector)

Method Detail

createWrappingDoFnRunner

protected org.apache.beam.runners.core.DoFnRunner<org.apache.beam.runners.core.KeyedWorkItem<byte[],org.apache.beam.sdk.values.KV<InputT,RestrictionT>>,OutputT> createWrappingDoFnRunner(org.apache.beam.runners.core.DoFnRunner<org.apache.beam.runners.core.KeyedWorkItem<byte[],org.apache.beam.sdk.values.KV<InputT,RestrictionT>>,OutputT> wrappedRunner,
                                                                                                                                                                                          org.apache.beam.runners.core.StepContext stepContext)

Overrides:

createWrappingDoFnRunner in class DoFnOperator<org.apache.beam.runners.core.KeyedWorkItem<byte[],org.apache.beam.sdk.values.KV<InputT,RestrictionT>>,OutputT>

initializeState

public void initializeState(org.apache.flink.runtime.state.StateInitializationContext context)
                     throws java.lang.Exception

Specified by:

initializeState in interface org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.CheckpointedStreamOperator

Overrides:

initializeState in class DoFnOperator<org.apache.beam.runners.core.KeyedWorkItem<byte[],org.apache.beam.sdk.values.KV<InputT,RestrictionT>>,OutputT>

Throws:

java.lang.Exception

fireTimer

protected void fireTimer(org.apache.beam.runners.core.TimerInternals.TimerData timer)

Overrides:

fireTimer in class DoFnOperator<org.apache.beam.runners.core.KeyedWorkItem<byte[],org.apache.beam.sdk.values.KV<InputT,RestrictionT>>,OutputT>

close

public void close()
           throws java.lang.Exception

Specified by:

close in interface org.apache.flink.streaming.api.operators.StreamOperator<org.apache.beam.sdk.util.WindowedValue<OutputT>>

Overrides:

close in class AbstractStreamOperatorCompat<org.apache.beam.sdk.util.WindowedValue<OutputT>>

Throws:

java.lang.Exception