org.apache.beam.runners.flink.translation.functions

Class AbstractFlinkCombineRunner<K,InputT,AccumT,OutputT,W extends org.apache.beam.sdk.transforms.windowing.BoundedWindow>

java.lang.Object

org.apache.beam.runners.flink.translation.functions.AbstractFlinkCombineRunner<K,InputT,AccumT,OutputT,W>

Direct Known Subclasses:

HashingFlinkCombineRunner, SingleWindowFlinkCombineRunner, SortingFlinkCombineRunner

public abstract class AbstractFlinkCombineRunner<K,InputT,AccumT,OutputT,W extends org.apache.beam.sdk.transforms.windowing.BoundedWindow>
extends java.lang.Object

Abstract base for runners that execute a Combine.PerKey. This unifies processing of merging/non-merging and partial/final combines.

The input to combine( FlinkCombiner, WindowingStrategy, SideInputReader, PipelineOptions, Iterable, Collector) are elements of the same key but * for different windows.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	AbstractFlinkCombineRunner.CompleteFlinkCombiner<K,InputT,AccumT,OutputT> A straight wrapper of CombineFnBase.GlobalCombineFn that takes in InputT and emits OutputT.
static class	AbstractFlinkCombineRunner.FinalFlinkCombiner<K,AccumT,OutputT> A final combiner that takes in AccumT and produces OutputT.
static interface	AbstractFlinkCombineRunner.FlinkCombiner<K,InputT,AccumT,OutputT> Adapter interface that allows using a CombineFnBase.GlobalCombineFn to either produce the AccumT as output or to combine several accumulators into an OutputT.
static class	AbstractFlinkCombineRunner.PartialFlinkCombiner<K,InputT,AccumT> A partial combiner that takes in InputT and produces AccumT.

Constructor Summary

Constructors

Constructor and Description
AbstractFlinkCombineRunner()

Method Summary

Modifier and Type

Method and Description

abstract void

combine(AbstractFlinkCombineRunner.FlinkCombiner<K,InputT,AccumT,OutputT> flinkCombiner, org.apache.beam.sdk.values.WindowingStrategy<java.lang.Object,W> windowingStrategy, org.apache.beam.runners.core.SideInputReader sideInputReader, org.apache.beam.sdk.options.PipelineOptions options, java.lang.Iterable<org.apache.beam.sdk.util.WindowedValue<org.apache.beam.sdk.values.KV<K,InputT>>> elements, org.apache.flink.util.Collector<org.apache.beam.sdk.util.WindowedValue<org.apache.beam.sdk.values.KV<K,OutputT>>> out) Consumes WindowedValues and produces combined output to the given output.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AbstractFlinkCombineRunner

public AbstractFlinkCombineRunner()

Method Detail

combine

public abstract void combine(AbstractFlinkCombineRunner.FlinkCombiner<K,InputT,AccumT,OutputT> flinkCombiner,
                             org.apache.beam.sdk.values.WindowingStrategy<java.lang.Object,W> windowingStrategy,
                             org.apache.beam.runners.core.SideInputReader sideInputReader,
                             org.apache.beam.sdk.options.PipelineOptions options,
                             java.lang.Iterable<org.apache.beam.sdk.util.WindowedValue<org.apache.beam.sdk.values.KV<K,InputT>>> elements,
                             org.apache.flink.util.Collector<org.apache.beam.sdk.util.WindowedValue<org.apache.beam.sdk.values.KV<K,OutputT>>> out)
                      throws java.lang.Exception

Consumes WindowedValues and produces combined output to the given output.

Throws:

java.lang.Exception