org.apache.beam.runners.samza.runtime

Interface BundleManager<OutT>

Type Parameters:

OutT - output type of the DoFnOp

All Known Implementing Classes:

ClassicBundleManager, PortableBundleManager

public interface BundleManager<OutT>

Bundle management for the DoFnOp that handles lifecycle of a bundle. It also serves as a proxy for the DoFnOp to process watermark and decides to 1. Hold watermark if there is at least one bundle in progress. 2. Propagates the watermark to downstream DAG, if all the previous bundles have completed.

A bundle is considered complete only when the outputs corresponding to each element in the bundle have been resolved and the watermark associated with the bundle(if any) is propagated downstream. The output of an element is considered resolved based on the nature of the ParDoFn 1. In case of synchronous ParDo, outputs of the element is resolved immediately after the processElement returns. 2. In case of asynchronous ParDo, outputs of the element is resolved when all the future emitted by the processElement is resolved.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static interface	BundleManager.BundleProgressListener<OutT> A listener used to track the lifecycle of a bundle.

Method Summary

Modifier and Type	Method and Description
void	processTimer(KeyedTimerData<java.lang.Void> keyedTimerData, OpEmitter<OutT> emitter) Signals the BundleManager that a timer is up.
void	processWatermark(org.joda.time.Instant watermark, OpEmitter<OutT> emitter) Signals a watermark event arrived.
void	signalFailure(java.lang.Throwable t) Fails the current bundle, throws away the pending output, and resets the bundle to an empty state.
void	tryFinishBundle(OpEmitter<OutT> emitter) Tries to close the bundle, and reset the bundle to an empty state.
void	tryStartBundle() Starts a new bundle if not already started, then adds an element to the existing bundle.

Method Detail

tryStartBundle

void tryStartBundle()

Starts a new bundle if not already started, then adds an element to the existing bundle.

processWatermark

void processWatermark(org.joda.time.Instant watermark,
                      OpEmitter<OutT> emitter)

Signals a watermark event arrived. The BundleManager will decide if the watermark needs to be processed, and notify the listener if needed.

Parameters:

watermark -

emitter -

processTimer

void processTimer(KeyedTimerData<java.lang.Void> keyedTimerData,
                  OpEmitter<OutT> emitter)

Signals the BundleManager that a timer is up.

Parameters:

keyedTimerData -

emitter -

signalFailure

void signalFailure(java.lang.Throwable t)

Fails the current bundle, throws away the pending output, and resets the bundle to an empty state.

Parameters:

t - the throwable that caused the failure.

tryFinishBundle

void tryFinishBundle(OpEmitter<OutT> emitter)

Tries to close the bundle, and reset the bundle to an empty state.

Parameters:

emitter -