org.apache.beam.runners.flink.translation.functions

Class FlinkPartialReduceFunction<K,InputT,AccumT,W extends org.apache.beam.sdk.transforms.windowing.BoundedWindow>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.api.common.functions.RichGroupCombineFunction<org.apache.beam.sdk.util.WindowedValue<org.apache.beam.sdk.values.KV<K,InputT>>,org.apache.beam.sdk.util.WindowedValue<org.apache.beam.sdk.values.KV<K,AccumT>>>

org.apache.beam.runners.flink.translation.functions.FlinkPartialReduceFunction<K,InputT,AccumT,W>

All Implemented Interfaces:

java.io.Serializable, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.GroupCombineFunction<org.apache.beam.sdk.util.WindowedValue<org.apache.beam.sdk.values.KV<K,InputT>>,org.apache.beam.sdk.util.WindowedValue<org.apache.beam.sdk.values.KV<K,AccumT>>>, org.apache.flink.api.common.functions.RichFunction

public class FlinkPartialReduceFunction<K,InputT,AccumT,W extends org.apache.beam.sdk.transforms.windowing.BoundedWindow>
extends org.apache.flink.api.common.functions.RichGroupCombineFunction<org.apache.beam.sdk.util.WindowedValue<org.apache.beam.sdk.values.KV<K,InputT>>,org.apache.beam.sdk.util.WindowedValue<org.apache.beam.sdk.values.KV<K,AccumT>>>

This is is the first step for executing a Combine.PerKey on Flink. The second part is FlinkReduceFunction. This function performs a local combine step before shuffling while the latter does the final combination after a shuffle.

The input to combine(Iterable, Collector) are elements of the same key but for different windows. We have to ensure that we only combine elements of matching windows.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.beam.sdk.transforms.CombineFnBase.GlobalCombineFn<InputT,AccumT,?>	combineFn
protected org.apache.beam.runners.core.construction.SerializablePipelineOptions	serializedOptions
protected java.util.Map<org.apache.beam.sdk.values.PCollectionView<?>,org.apache.beam.sdk.values.WindowingStrategy<?,?>>	sideInputs
protected org.apache.beam.sdk.values.WindowingStrategy<java.lang.Object,W>	windowingStrategy

Constructor Summary

Constructors

Constructor and Description
FlinkPartialReduceFunction(org.apache.beam.sdk.transforms.CombineFnBase.GlobalCombineFn<InputT,AccumT,?> combineFn, org.apache.beam.sdk.values.WindowingStrategy<java.lang.Object,W> windowingStrategy, java.util.Map<org.apache.beam.sdk.values.PCollectionView<?>,org.apache.beam.sdk.values.WindowingStrategy<?,?>> sideInputs, org.apache.beam.sdk.options.PipelineOptions pipelineOptions)
FlinkPartialReduceFunction(org.apache.beam.sdk.transforms.CombineFnBase.GlobalCombineFn<InputT,AccumT,?> combineFn, org.apache.beam.sdk.values.WindowingStrategy<java.lang.Object,W> windowingStrategy, java.util.Map<org.apache.beam.sdk.values.PCollectionView<?>,org.apache.beam.sdk.values.WindowingStrategy<?,?>> sideInputs, org.apache.beam.sdk.options.PipelineOptions pipelineOptions, boolean groupedByWindow)

Method Summary

Modifier and Type	Method and Description
void	combine(java.lang.Iterable<org.apache.beam.sdk.util.WindowedValue<org.apache.beam.sdk.values.KV<K,InputT>>> elements, org.apache.flink.util.Collector<org.apache.beam.sdk.util.WindowedValue<org.apache.beam.sdk.values.KV<K,AccumT>>> out)
void	open(org.apache.flink.configuration.Configuration parameters)

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

close, getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

combineFn

protected final org.apache.beam.sdk.transforms.CombineFnBase.GlobalCombineFn<InputT,AccumT,?> combineFn

windowingStrategy

protected final org.apache.beam.sdk.values.WindowingStrategy<java.lang.Object,W extends org.apache.beam.sdk.transforms.windowing.BoundedWindow> windowingStrategy

serializedOptions

protected final org.apache.beam.runners.core.construction.SerializablePipelineOptions serializedOptions

sideInputs

protected final java.util.Map<org.apache.beam.sdk.values.PCollectionView<?>,org.apache.beam.sdk.values.WindowingStrategy<?,?>> sideInputs

Constructor Detail

FlinkPartialReduceFunction

public FlinkPartialReduceFunction(org.apache.beam.sdk.transforms.CombineFnBase.GlobalCombineFn<InputT,AccumT,?> combineFn,
                                  org.apache.beam.sdk.values.WindowingStrategy<java.lang.Object,W> windowingStrategy,
                                  java.util.Map<org.apache.beam.sdk.values.PCollectionView<?>,org.apache.beam.sdk.values.WindowingStrategy<?,?>> sideInputs,
                                  org.apache.beam.sdk.options.PipelineOptions pipelineOptions)

FlinkPartialReduceFunction

public FlinkPartialReduceFunction(org.apache.beam.sdk.transforms.CombineFnBase.GlobalCombineFn<InputT,AccumT,?> combineFn,
                                  org.apache.beam.sdk.values.WindowingStrategy<java.lang.Object,W> windowingStrategy,
                                  java.util.Map<org.apache.beam.sdk.values.PCollectionView<?>,org.apache.beam.sdk.values.WindowingStrategy<?,?>> sideInputs,
                                  org.apache.beam.sdk.options.PipelineOptions pipelineOptions,
                                  boolean groupedByWindow)

Method Detail

open

public void open(org.apache.flink.configuration.Configuration parameters)

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

open in class org.apache.flink.api.common.functions.AbstractRichFunction

combine

public void combine(java.lang.Iterable<org.apache.beam.sdk.util.WindowedValue<org.apache.beam.sdk.values.KV<K,InputT>>> elements,
                    org.apache.flink.util.Collector<org.apache.beam.sdk.util.WindowedValue<org.apache.beam.sdk.values.KV<K,AccumT>>> out)
             throws java.lang.Exception

Specified by:

combine in interface org.apache.flink.api.common.functions.GroupCombineFunction<org.apache.beam.sdk.util.WindowedValue<org.apache.beam.sdk.values.KV<K,InputT>>,org.apache.beam.sdk.util.WindowedValue<org.apache.beam.sdk.values.KV<K,AccumT>>>

Specified by:

combine in class org.apache.flink.api.common.functions.RichGroupCombineFunction<org.apache.beam.sdk.util.WindowedValue<org.apache.beam.sdk.values.KV<K,InputT>>,org.apache.beam.sdk.util.WindowedValue<org.apache.beam.sdk.values.KV<K,AccumT>>>

Throws:

java.lang.Exception