org.apache.beam.runners.flink.translation.wrappers.streaming.io

Class UnboundedSourceWrapper<OutputT,CheckpointMarkT extends org.apache.beam.sdk.io.UnboundedSource.CheckpointMark>

java.lang.Object

org.apache.flink.api.common.functions.AbstractRichFunction

org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction<org.apache.beam.sdk.util.WindowedValue<org.apache.beam.sdk.values.ValueWithRecordId<OutputT>>>

org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper<OutputT,CheckpointMarkT>

All Implemented Interfaces:

java.io.Serializable, BeamStoppableFunction, org.apache.flink.api.common.functions.Function, org.apache.flink.api.common.functions.RichFunction, org.apache.flink.runtime.state.CheckpointListener, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction, org.apache.flink.streaming.api.functions.source.ParallelSourceFunction<org.apache.beam.sdk.util.WindowedValue<org.apache.beam.sdk.values.ValueWithRecordId<OutputT>>>, org.apache.flink.streaming.api.functions.source.SourceFunction<org.apache.beam.sdk.util.WindowedValue<org.apache.beam.sdk.values.ValueWithRecordId<OutputT>>>, org.apache.flink.streaming.runtime.tasks.ProcessingTimeCallback

public class UnboundedSourceWrapper<OutputT,CheckpointMarkT extends org.apache.beam.sdk.io.UnboundedSource.CheckpointMark>
extends org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction<org.apache.beam.sdk.util.WindowedValue<org.apache.beam.sdk.values.ValueWithRecordId<OutputT>>>
implements org.apache.flink.streaming.runtime.tasks.ProcessingTimeCallback, BeamStoppableFunction, org.apache.flink.runtime.state.CheckpointListener, org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

Wrapper for executing UnboundedSources as a Flink Source.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flink.streaming.api.functions.source.SourceFunction

org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<T>

Constructor Summary

Constructors

Constructor and Description
UnboundedSourceWrapper(java.lang.String stepName, org.apache.beam.sdk.options.PipelineOptions pipelineOptions, org.apache.beam.sdk.io.UnboundedSource<OutputT,CheckpointMarkT> source, int parallelism)

Method Summary

Modifier and Type	Method and Description
void	cancel()
void	close()
java.util.List<? extends org.apache.beam.sdk.io.UnboundedSource<OutputT,CheckpointMarkT>>	getSplitSources() Visible so that we can check this in tests.
void	initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
void	notifyCheckpointComplete(long checkpointId)
void	onProcessingTime(long timestamp)
void	open(org.apache.flink.configuration.Configuration parameters) Initialize and restore state before starting execution of the source.
void	run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<org.apache.beam.sdk.util.WindowedValue<org.apache.beam.sdk.values.ValueWithRecordId<OutputT>>> ctx)
void	setSourceContext(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<org.apache.beam.sdk.util.WindowedValue<org.apache.beam.sdk.values.ValueWithRecordId<OutputT>>> ctx) Visible so that we can set this in tests.
void	snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext functionSnapshotContext)
void	stop() Unused method for backward compatibility.

Methods inherited from class org.apache.flink.api.common.functions.AbstractRichFunction

getIterationRuntimeContext, getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.runtime.state.CheckpointListener

notifyCheckpointAborted

Constructor Detail

UnboundedSourceWrapper

public UnboundedSourceWrapper(java.lang.String stepName,
                              org.apache.beam.sdk.options.PipelineOptions pipelineOptions,
                              org.apache.beam.sdk.io.UnboundedSource<OutputT,CheckpointMarkT> source,
                              int parallelism)
                       throws java.lang.Exception

Throws:

java.lang.Exception

Method Detail

open

public void open(org.apache.flink.configuration.Configuration parameters)
          throws java.lang.Exception

Initialize and restore state before starting execution of the source.

Specified by:

open in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

open in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

java.lang.Exception

run

public void run(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<org.apache.beam.sdk.util.WindowedValue<org.apache.beam.sdk.values.ValueWithRecordId<OutputT>>> ctx)
         throws java.lang.Exception

Specified by:

run in interface org.apache.flink.streaming.api.functions.source.SourceFunction<org.apache.beam.sdk.util.WindowedValue<org.apache.beam.sdk.values.ValueWithRecordId<OutputT>>>

Throws:

java.lang.Exception

close

public void close()
           throws java.lang.Exception

Specified by:

close in interface org.apache.flink.api.common.functions.RichFunction

Overrides:

close in class org.apache.flink.api.common.functions.AbstractRichFunction

Throws:

java.lang.Exception

cancel

public void cancel()

Specified by:

cancel in interface org.apache.flink.streaming.api.functions.source.SourceFunction<org.apache.beam.sdk.util.WindowedValue<org.apache.beam.sdk.values.ValueWithRecordId<OutputT>>>

stop

public void stop()

Description copied from interface: BeamStoppableFunction

Unused method for backward compatibility.

Specified by:

stop in interface BeamStoppableFunction

snapshotState

public void snapshotState(org.apache.flink.runtime.state.FunctionSnapshotContext functionSnapshotContext)
                   throws java.lang.Exception

Specified by:

snapshotState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

Throws:

java.lang.Exception

initializeState

public void initializeState(org.apache.flink.runtime.state.FunctionInitializationContext context)
                     throws java.lang.Exception

Specified by:

initializeState in interface org.apache.flink.streaming.api.checkpoint.CheckpointedFunction

Throws:

java.lang.Exception

onProcessingTime

public void onProcessingTime(long timestamp)

Specified by:

onProcessingTime in interface org.apache.flink.streaming.runtime.tasks.ProcessingTimeCallback

getSplitSources

public java.util.List<? extends org.apache.beam.sdk.io.UnboundedSource<OutputT,CheckpointMarkT>> getSplitSources()

Visible so that we can check this in tests. Must not be used for anything else.

setSourceContext

public void setSourceContext(org.apache.flink.streaming.api.functions.source.SourceFunction.SourceContext<org.apache.beam.sdk.util.WindowedValue<org.apache.beam.sdk.values.ValueWithRecordId<OutputT>>> ctx)

Visible so that we can set this in tests. This is only set in the run method which is inconvenient for the tests where the context is assumed to be set when run is called. Must not be used for anything else.

notifyCheckpointComplete

public void notifyCheckpointComplete(long checkpointId)
                              throws java.lang.Exception

Specified by:

notifyCheckpointComplete in interface org.apache.flink.runtime.state.CheckpointListener

Throws:

java.lang.Exception