org.apache.beam.runners.flink.translation.wrappers.streaming

Class DoFnOperator.BufferedOutputManager<OutputT>

java.lang.Object

org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.BufferedOutputManager<OutputT>

All Implemented Interfaces:

org.apache.beam.runners.core.DoFnRunners.OutputManager

Enclosing class:

DoFnOperator<InputT,OutputT>

public static class DoFnOperator.BufferedOutputManager<OutputT>
extends java.lang.Object
implements org.apache.beam.runners.core.DoFnRunners.OutputManager

A DoFnRunners.OutputManager that can buffer its outputs. Uses PushedBackElementsHandler to buffer the data. Buffering data is necessary because no elements can be emitted during snapshotState which is called when the checkpoint barrier already has been sent downstream. Emitting elements would break the flow of checkpoint barrier and violate exactly-once semantics.

This buffering can be deactived using FlinkPipelineOptions#setFinishBundleBeforeCheckpointing(true). If activated, we flush out bundle data before the barrier is sent downstream. This is done via prepareSnapshotPreBarrier. When Flink supports unaligned checkpoints, this should become the default and this class should be removed as in https://github.com/apache/beam/pull/9652.

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.flink.streaming.api.operators.Output<org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.beam.sdk.util.WindowedValue<OutputT>>>	output

Method Summary

Modifier and Type	Method and Description
<T> void	output(org.apache.beam.sdk.values.TupleTag<T> tag, org.apache.beam.sdk.util.WindowedValue<T> value)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

output

protected final org.apache.flink.streaming.api.operators.Output<org.apache.flink.streaming.runtime.streamrecord.StreamRecord<org.apache.beam.sdk.util.WindowedValue<OutputT>>> output

Method Detail

output

public <T> void output(org.apache.beam.sdk.values.TupleTag<T> tag,
                       org.apache.beam.sdk.util.WindowedValue<T> value)

Specified by:

output in interface org.apache.beam.runners.core.DoFnRunners.OutputManager