org.apache.beam.fn.harness.control

Class ProcessBundleHandler

java.lang.Object

org.apache.beam.fn.harness.control.ProcessBundleHandler

public class ProcessBundleHandler
extends java.lang.Object

Processes BeamFnApi.ProcessBundleRequests and BeamFnApi.ProcessBundleSplitRequests.

BeamFnApi.ProcessBundleSplitRequests use a cache to find/create a ProcessBundleHandler.BundleProcessor. The creation of a ProcessBundleHandler.BundleProcessor uses the associated BeamFnApi.ProcessBundleDescriptor definition; creating runners for each RunnerApi.FunctionSpec; wiring them together based upon the input and output map definitions. The ProcessBundleHandler.BundleProcessor executes the DAG based graph by starting all runners in reverse topological order, and finishing all runners in forward topological order.

BeamFnApi.ProcessBundleSplitRequests finds an active ProcessBundleHandler.BundleProcessor associated with a currently processing BeamFnApi.ProcessBundleRequest and uses it to perform a split request. See breaking the fusion barrier for further details.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ProcessBundleHandler.BundleProcessor A container for the reusable information used to process a bundle.
static class	ProcessBundleHandler.BundleProcessorCache A cache for ProcessBundleHandler.BundleProcessors.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	JAVA_SOURCE_URN

Constructor Summary

Constructors

Constructor and Description

ProcessBundleHandler(org.apache.beam.sdk.options.PipelineOptions options, java.util.Set<java.lang.String> runnerCapabilities, java.util.function.Function<java.lang.String,org.apache.beam.model.fnexecution.v1.BeamFnApi.ProcessBundleDescriptor> fnApiRegistry, BeamFnDataClient beamFnDataClient, BeamFnStateGrpcClientCache beamFnStateGrpcClientCache, FinalizeBundleHandler finalizeBundleHandler, org.apache.beam.runners.core.metrics.ShortIdMap shortIds, ExecutionStateSampler executionStateSampler, Cache<java.lang.Object,java.lang.Object> processWideCache, @Nullable DataSampler dataSampler)

Method Summary

Modifier and Type	Method and Description
ProcessBundleHandler.BundleProcessorCache	getBundleProcessorCache()
org.apache.beam.model.fnexecution.v1.BeamFnApi.InstructionResponse.Builder	processBundle(org.apache.beam.model.fnexecution.v1.BeamFnApi.InstructionRequest request) Processes a bundle, running the start(), process(), and finish() functions.
org.apache.beam.model.fnexecution.v1.BeamFnApi.InstructionResponse.Builder	progress(org.apache.beam.model.fnexecution.v1.BeamFnApi.InstructionRequest request)
void	shutdown() Shutdown the bundles, running the tearDown() functions.
org.apache.beam.model.fnexecution.v1.BeamFnApi.InstructionResponse.Builder	trySplit(org.apache.beam.model.fnexecution.v1.BeamFnApi.InstructionRequest request) Splits an active bundle.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

JAVA_SOURCE_URN

public static final java.lang.String JAVA_SOURCE_URN

See Also:

Constant Field Values

Constructor Detail

ProcessBundleHandler

public ProcessBundleHandler(org.apache.beam.sdk.options.PipelineOptions options,
                            java.util.Set<java.lang.String> runnerCapabilities,
                            java.util.function.Function<java.lang.String,org.apache.beam.model.fnexecution.v1.BeamFnApi.ProcessBundleDescriptor> fnApiRegistry,
                            BeamFnDataClient beamFnDataClient,
                            BeamFnStateGrpcClientCache beamFnStateGrpcClientCache,
                            FinalizeBundleHandler finalizeBundleHandler,
                            org.apache.beam.runners.core.metrics.ShortIdMap shortIds,
                            ExecutionStateSampler executionStateSampler,
                            Cache<java.lang.Object,java.lang.Object> processWideCache,
                            @Nullable DataSampler dataSampler)

Method Detail

processBundle

public org.apache.beam.model.fnexecution.v1.BeamFnApi.InstructionResponse.Builder processBundle(org.apache.beam.model.fnexecution.v1.BeamFnApi.InstructionRequest request)
                                                                                         throws java.lang.Exception

Processes a bundle, running the start(), process(), and finish() functions. This function is required to be reentrant.

Throws:

java.lang.Exception

progress

public org.apache.beam.model.fnexecution.v1.BeamFnApi.InstructionResponse.Builder progress(org.apache.beam.model.fnexecution.v1.BeamFnApi.InstructionRequest request)
                                                                                    throws java.lang.Exception

Throws:

java.lang.Exception

trySplit

public org.apache.beam.model.fnexecution.v1.BeamFnApi.InstructionResponse.Builder trySplit(org.apache.beam.model.fnexecution.v1.BeamFnApi.InstructionRequest request)

Splits an active bundle.

shutdown

public void shutdown()
              throws java.lang.Exception

Shutdown the bundles, running the tearDown() functions.

Throws:

java.lang.Exception

getBundleProcessorCache

public ProcessBundleHandler.BundleProcessorCache getBundleProcessorCache()