org.apache.beam.runners.flink.translation.wrappers.streaming.stableinput

Class BufferingDoFnRunner<InputT,OutputT>

java.lang.Object

org.apache.beam.runners.flink.translation.wrappers.streaming.stableinput.BufferingDoFnRunner<InputT,OutputT>

All Implemented Interfaces:

org.apache.beam.runners.core.DoFnRunner<InputT,OutputT>

public class BufferingDoFnRunner<InputT,OutputT>
extends java.lang.Object
implements org.apache.beam.runners.core.DoFnRunner<InputT,OutputT>

A DoFnRunner which buffers data for supporting DoFn.RequiresStableInput.

When a DoFn is annotated with @RequiresStableInput we are only allowed to process elements after a checkpoint has completed. This ensures that the input is stable and we produce idempotent results on failures.

Method Summary

Modifier and Type	Method and Description
void	checkpoint(long checkpointId) Should be called when a checkpoint is created.
void	checkpointCompleted(long checkpointId) Should be called when a checkpoint is completed.
static <InputT,OutputT> BufferingDoFnRunner<InputT,OutputT>	create(org.apache.beam.runners.core.DoFnRunner<InputT,OutputT> doFnRunner, java.lang.String stateName, org.apache.beam.sdk.coders.Coder windowedInputCoder, org.apache.beam.sdk.coders.Coder windowCoder, org.apache.flink.runtime.state.OperatorStateBackend operatorStateBackend, @Nullable org.apache.flink.runtime.state.KeyedStateBackend<java.lang.Object> keyedStateBackend, int maxConcurrentCheckpoints, org.apache.beam.runners.core.construction.SerializablePipelineOptions pipelineOptions)
void	finishBundle()
org.apache.beam.sdk.transforms.DoFn<InputT,OutputT>	getFn()
<KeyT> void	onTimer(java.lang.String timerId, java.lang.String timerFamilyId, KeyT key, org.apache.beam.sdk.transforms.windowing.BoundedWindow window, org.joda.time.Instant timestamp, org.joda.time.Instant outputTimestamp, org.apache.beam.sdk.state.TimeDomain timeDomain)
<KeyT> void	onWindowExpiration(org.apache.beam.sdk.transforms.windowing.BoundedWindow window, org.joda.time.Instant timestamp, KeyT key)
void	processElement(org.apache.beam.sdk.util.WindowedValue<InputT> elem)
void	startBundle()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

create

public static <InputT,OutputT> BufferingDoFnRunner<InputT,OutputT> create(org.apache.beam.runners.core.DoFnRunner<InputT,OutputT> doFnRunner,
                                                                          java.lang.String stateName,
                                                                          org.apache.beam.sdk.coders.Coder windowedInputCoder,
                                                                          org.apache.beam.sdk.coders.Coder windowCoder,
                                                                          org.apache.flink.runtime.state.OperatorStateBackend operatorStateBackend,
                                                                          @Nullable org.apache.flink.runtime.state.KeyedStateBackend<java.lang.Object> keyedStateBackend,
                                                                          int maxConcurrentCheckpoints,
                                                                          org.apache.beam.runners.core.construction.SerializablePipelineOptions pipelineOptions)
                                                                   throws java.lang.Exception

Throws:

java.lang.Exception

startBundle

public void startBundle()

Specified by:

startBundle in interface org.apache.beam.runners.core.DoFnRunner<InputT,OutputT>

processElement

public void processElement(org.apache.beam.sdk.util.WindowedValue<InputT> elem)

Specified by:

processElement in interface org.apache.beam.runners.core.DoFnRunner<InputT,OutputT>

onTimer

public <KeyT> void onTimer(java.lang.String timerId,
                           java.lang.String timerFamilyId,
                           KeyT key,
                           org.apache.beam.sdk.transforms.windowing.BoundedWindow window,
                           org.joda.time.Instant timestamp,
                           org.joda.time.Instant outputTimestamp,
                           org.apache.beam.sdk.state.TimeDomain timeDomain)

Specified by:

onTimer in interface org.apache.beam.runners.core.DoFnRunner<InputT,OutputT>

finishBundle

public void finishBundle()

Specified by:

finishBundle in interface org.apache.beam.runners.core.DoFnRunner<InputT,OutputT>

onWindowExpiration

public <KeyT> void onWindowExpiration(org.apache.beam.sdk.transforms.windowing.BoundedWindow window,
                                      org.joda.time.Instant timestamp,
                                      KeyT key)

Specified by:

onWindowExpiration in interface org.apache.beam.runners.core.DoFnRunner<InputT,OutputT>

getFn

public org.apache.beam.sdk.transforms.DoFn<InputT,OutputT> getFn()

Specified by:

getFn in interface org.apache.beam.runners.core.DoFnRunner<InputT,OutputT>

checkpoint

public void checkpoint(long checkpointId)
                throws java.lang.Exception

Should be called when a checkpoint is created.

Throws:

java.lang.Exception

checkpointCompleted

public void checkpointCompleted(long checkpointId)
                         throws java.lang.Exception

Should be called when a checkpoint is completed.

Throws:

java.lang.Exception