Index (3 2.61.0 API)

A B C D E F G H I J K L M N O P R S T U V W
All Classes All Packages

A

AccumulatorCheckpointingSparkListener() - Constructor for class org.apache.beam.runners.spark.metrics.MetricsAccumulator.AccumulatorCheckpointingSparkListener
action() - Method in class org.apache.beam.runners.spark.translation.BoundedDataset
action() - Method in interface org.apache.beam.runners.spark.translation.Dataset
action() - Method in class org.apache.beam.runners.spark.translation.streaming.UnboundedDataset
add(int, GlobalWatermarkHolder.SparkWatermarks) - Static method in class org.apache.beam.runners.spark.util.GlobalWatermarkHolder
add(MetricsContainerStepMap) - Method in class org.apache.beam.runners.spark.metrics.MetricsContainerStepMapAccumulator
add(MetricsContainerStepMap) - Method in class org.apache.beam.runners.spark.structuredstreaming.metrics.MetricsAccumulator
add(WindowedValue<InputT>, SparkCombineFn<InputT, ValueT, AccumT, ?>) - Method in interface org.apache.beam.runners.spark.translation.SparkCombineFn.WindowedAccumulator: Add value with unexploded windows into the accumulator.
addAll(Map<Integer, Queue<GlobalWatermarkHolder.SparkWatermarks>>) - Static method in class org.apache.beam.runners.spark.util.GlobalWatermarkHolder
advance() - Method in class org.apache.beam.runners.spark.io.MicrobatchSource.Reader
advance() - Static method in class org.apache.beam.runners.spark.util.GlobalWatermarkHolder: See GlobalWatermarkHolder.advance(String).
advanceNextBatchWatermarkToInfinity() - Method in class org.apache.beam.runners.spark.io.CreateStream: Advances the watermark in the next batch to the end-of-time.
advanceWatermark() - Method in class org.apache.beam.runners.spark.stateful.SparkTimerInternals: Advances the watermark.
advanceWatermarkForNextBatch(Instant) - Method in class org.apache.beam.runners.spark.io.CreateStream: Advances the watermark in the next batch.
apply(KV<String, Long>) - Method in class org.apache.beam.runners.spark.examples.WordCount.FormatAsTextFn
apply(KV<String, Long>) - Method in class org.apache.beam.runners.spark.structuredstreaming.examples.WordCount.FormatAsTextFn
apply(Tuple2<ByteArray, byte[]>) - Method in class org.apache.beam.runners.spark.coders.CoderHelpers.FromByteFunction
awaitTermination(Duration) - Method in class org.apache.beam.runners.spark.SparkPipelineResult
awaitTermination(Duration) - Method in class org.apache.beam.runners.spark.SparkRunnerDebugger.DebugSparkPipelineResult

B

borrowDataset(PTransform<? extends PValue, ?>) - Method in class org.apache.beam.runners.spark.translation.EvaluationContext
borrowDataset(PValue) - Method in class org.apache.beam.runners.spark.translation.EvaluationContext
Bounded(SparkContext, BoundedSource<T>, SerializablePipelineOptions, String) - Constructor for class org.apache.beam.runners.spark.io.SourceRDD.Bounded
BoundedDataset<T> - Class in org.apache.beam.runners.spark.translation: Holds an RDD or values for deferred conversion to an RDD if needed.
BoundedDatasetFactory - Class in org.apache.beam.runners.spark.structuredstreaming.io
broadcast(JavaSparkContext) - Method in class org.apache.beam.runners.spark.util.SideInputBroadcast
broadcast(T) - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator.Context
ByteArray - Class in org.apache.beam.runners.spark.util: Serializable byte array.
ByteArray(byte[]) - Constructor for class org.apache.beam.runners.spark.util.ByteArray

C

cache(String, Coder<?>) - Method in class org.apache.beam.runners.spark.translation.BoundedDataset
cache(String, Coder<?>) - Method in interface org.apache.beam.runners.spark.translation.Dataset
cache(String, Coder<?>) - Method in class org.apache.beam.runners.spark.translation.streaming.UnboundedDataset
CachedSideInputReader - Class in org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions: SideInputReader that caches results for costly Materializations.
CachedSideInputReader - Class in org.apache.beam.runners.spark.util: SideInputReader that caches materialized views.
call() - Method in class org.apache.beam.runners.spark.translation.streaming.SparkRunnerStreamingContextFactory
call(Iterator<WindowedValue<InputT>>) - Method in class org.apache.beam.runners.spark.translation.MultiDoFnFunction
call(K, Iterator<WindowedValue<KV<K, InputT>>>) - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions.GroupAlsoByWindowViaOutputBufferFn
call(WindowedValue<KV<K, Iterable<InputT>>>) - Method in class org.apache.beam.runners.spark.translation.TranslationUtils.CombineGroupedValues
call(WindowedValue<KV<K, V>>) - Method in class org.apache.beam.runners.spark.translation.ReifyTimestampsAndWindowsFunction
call(WindowedValue<T>) - Method in class org.apache.beam.runners.spark.translation.SparkAssignWindowFn
call(Tuple2<ByteArray, byte[]>) - Method in class org.apache.beam.runners.spark.coders.CoderHelpers.FromByteFunction
call(Tuple2<TupleTag<V>, WindowedValue<?>>) - Method in class org.apache.beam.runners.spark.translation.TranslationUtils.TupleTagFilter
cancel() - Method in class org.apache.beam.runners.spark.SparkPipelineResult
cancel() - Method in class org.apache.beam.runners.spark.structuredstreaming.SparkStructuredStreamingPipelineResult
canTranslate(TransformT) - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator: Checks if a composite / primitive transform can be translated.
Checkpoint - Class in org.apache.beam.runners.spark.translation.streaming: Checkpoint data to make it available in future pipeline runs.
Checkpoint() - Constructor for class org.apache.beam.runners.spark.translation.streaming.Checkpoint
Checkpoint.CheckpointDir - Class in org.apache.beam.runners.spark.translation.streaming: Checkpoint dir tree.
CheckpointDir(String) - Constructor for class org.apache.beam.runners.spark.translation.streaming.Checkpoint.CheckpointDir
clear() - Static method in class org.apache.beam.runners.spark.metrics.MetricsAccumulator
clear() - Static method in class org.apache.beam.runners.spark.structuredstreaming.metrics.MetricsAccumulator
clear() - Static method in class org.apache.beam.runners.spark.util.GlobalWatermarkHolder
clearCache() - Static method in class org.apache.beam.runners.spark.io.MicrobatchSource
clearProvidedSparkContext() - Static method in class org.apache.beam.runners.spark.translation.SparkContextFactory
close() - Method in class org.apache.beam.runners.spark.io.MicrobatchSource.Reader
CodahaleCsvSink - Class in org.apache.beam.runners.spark.structuredstreaming.metrics.sink: A Sink for Spark's metric system reporting metrics (including Beam step metrics) to a CSV file.
CodahaleCsvSink(Properties, MetricRegistry) - Constructor for class org.apache.beam.runners.spark.structuredstreaming.metrics.sink.CodahaleCsvSink: Constructor for Spark 3.2.x and later.
CodahaleCsvSink(Properties, MetricRegistry, SecurityManager) - Constructor for class org.apache.beam.runners.spark.structuredstreaming.metrics.sink.CodahaleCsvSink: Constructor for Spark 3.1.x and earlier.
CodahaleGraphiteSink - Class in org.apache.beam.runners.spark.structuredstreaming.metrics.sink: A Sink for Spark's metric system reporting metrics (including Beam step metrics) to Graphite.
CodahaleGraphiteSink(Properties, MetricRegistry) - Constructor for class org.apache.beam.runners.spark.structuredstreaming.metrics.sink.CodahaleGraphiteSink: Constructor for Spark 3.2.x and later.
CodahaleGraphiteSink(Properties, MetricRegistry, SecurityManager) - Constructor for class org.apache.beam.runners.spark.structuredstreaming.metrics.sink.CodahaleGraphiteSink: Constructor for Spark 3.1.x and earlier.
CoderHelpers - Class in org.apache.beam.runners.spark.coders: Serialization utility class.
CoderHelpers - Class in org.apache.beam.runners.spark.structuredstreaming.translation.helpers: Serialization utility class.
CoderHelpers.FromByteFunction<K,V> - Class in org.apache.beam.runners.spark.coders: A function for converting a byte array pair to a key-value pair.
collect(String, Dataset<T>) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.EvaluationContext: The purpose of this utility is to mark the evaluation of Spark actions, both during Pipeline translation, when evaluation is required, and when finally evaluating the pipeline.
collectionEncoder(Encoder<T>) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderHelpers: Creates a Spark Encoder of ArrayType for Java Collections with nullable elements.
collectionEncoder(Encoder<T>, boolean) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderHelpers: Creates a Spark Encoder of ArrayType for Java Collections.
combineGlobally(JavaRDD<WindowedValue<InputT>>, SparkCombineFn<InputT, InputT, AccumT, OutputT>, Coder<AccumT>, WindowingStrategy<?, ?>) - Static method in class org.apache.beam.runners.spark.translation.GroupCombineFunctions: Apply a composite Combine.Globally transformation.
CombineGroupedValues(SparkCombineFn<KV<K, InputT>, InputT, ?, OutputT>) - Constructor for class org.apache.beam.runners.spark.translation.TranslationUtils.CombineGroupedValues
combinePerKey(JavaRDD<WindowedValue<KV<K, V>>>, SparkCombineFn<KV<K, V>, V, AccumT, ?>, Coder<K>, Coder<V>, Coder<AccumT>, WindowingStrategy<?, ?>) - Static method in class org.apache.beam.runners.spark.translation.GroupCombineFunctions: Apply a composite Combine.PerKey transformation.
compareTo(ByteArray) - Method in class org.apache.beam.runners.spark.util.ByteArray
complexityFactor - Variable in class org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator
compute(Partition, TaskContext) - Method in class org.apache.beam.runners.spark.io.SourceRDD.Bounded
compute(Partition, TaskContext) - Method in class org.apache.beam.runners.spark.io.SourceRDD.Unbounded
compute(Time) - Method in class org.apache.beam.runners.spark.translation.streaming.TestDStream
computeOutputs() - Method in class org.apache.beam.runners.spark.translation.EvaluationContext: Computes the outputs for all RDDs that are leaves in the DAG and do not have any actions (like saving to a file) registered on them (i.e.
computeOutputs() - Method in class org.apache.beam.runners.spark.translation.SparkTranslationContext: Compute the outputs for all RDDs that are leaves in the DAG.
concat(List<T>, List<T>) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.utils.ScalaInterop
ConsoleIO - Class in org.apache.beam.runners.spark.io: Print to console.
ConsoleIO.Write - Class in org.apache.beam.runners.spark.io: Write to console.
ConsoleIO.Write.Unbound<T> - Class in org.apache.beam.runners.spark.io: PTransform writing PCollection to the console.
contains(PCollectionView<T>) - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions.CachedSideInputReader
contains(PCollectionView<T>) - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions.SparkSideInputReader
contains(PCollectionView<T>) - Method in class org.apache.beam.runners.spark.util.CachedSideInputReader
contains(PCollectionView<T>) - Method in class org.apache.beam.runners.spark.util.SparkSideInputReader
copy() - Method in class org.apache.beam.runners.spark.metrics.MetricsContainerStepMapAccumulator
copy() - Method in class org.apache.beam.runners.spark.structuredstreaming.metrics.MetricsAccumulator
copyToList(ArrayData, DataType) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderHelpers.Utils
CountWords() - Constructor for class org.apache.beam.runners.spark.examples.WordCount.CountWords
CountWords() - Constructor for class org.apache.beam.runners.spark.structuredstreaming.examples.WordCount.CountWords
create() - Static method in class org.apache.beam.runners.spark.SparkRunner: Creates and returns a new SparkRunner with default options.
create() - Static method in class org.apache.beam.runners.spark.structuredstreaming.SparkStructuredStreamingRunner: Creates and returns a new SparkStructuredStreamingRunner with default options.
create(byte[], Coder<T>) - Static method in class org.apache.beam.runners.spark.util.SideInputBroadcast
create(Map<String, Broadcast<SideInputValues<?>>>) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions.SparkSideInputReader: Creates a SideInputReader for Spark from a map of PCollectionView tag ids and the corresponding broadcasted SideInputValues.
create(SparkJobServerDriver.SparkServerConfiguration) - Static method in class org.apache.beam.runners.spark.SparkJobInvoker
create(SparkPipelineOptions) - Static method in class org.apache.beam.runners.spark.SparkRunner: Creates and returns a new SparkRunner with specified options.
create(SparkStructuredStreamingPipelineOptions) - Static method in class org.apache.beam.runners.spark.structuredstreaming.SparkStructuredStreamingRunner: Creates and returns a new SparkStructuredStreamingRunner with specified options.
create(SparkCombineFn<InputT, ValueT, AccumT, ?>, Function<InputT, ValueT>, WindowingStrategy<?, ?>, Comparator<BoundedWindow>) - Static method in interface org.apache.beam.runners.spark.translation.SparkCombineFn.WindowedAccumulator: Create concrete accumulator for given type.
create(PipelineOptions) - Method in class org.apache.beam.runners.spark.SparkCommonPipelineOptions.StorageLevelFactory
create(PipelineOptions) - Method in class org.apache.beam.runners.spark.SparkCommonPipelineOptions.TmpCheckpointDirFactory
create(PipelineOptions) - Method in class org.apache.beam.runners.spark.SparkContextOptions.EmptyListenersList
create(PipelineOptions) - Method in class org.apache.beam.runners.spark.TestSparkPipelineOptions.DefaultStopPipelineWatermarkFactory
create(Function<InputT, ValueT>, SparkCombineFn.WindowedAccumulator.Type, Iterable<WindowedValue<AccumT>>, Comparator<BoundedWindow>) - Static method in interface org.apache.beam.runners.spark.translation.SparkCombineFn.WindowedAccumulator: Create concrete accumulator for given type.
create(Function<InputT, ValueT>, SparkCombineFn.WindowedAccumulator.Type, Comparator<BoundedWindow>) - Static method in interface org.apache.beam.runners.spark.translation.SparkCombineFn.WindowedAccumulator
createDataset(List<WindowedValue<T>>, Encoder<WindowedValue<T>>) - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator.Context
createDatasetFromRDD(SparkSession, BoundedSource<T>, Supplier<PipelineOptions>, Encoder<WindowedValue<T>>) - Static method in class org.apache.beam.runners.spark.structuredstreaming.io.BoundedDatasetFactory: Create a Dataset for a BoundedSource via a Spark RDD.
createDatasetFromRows(SparkSession, BoundedSource<T>, Supplier<PipelineOptions>, Encoder<WindowedValue<T>>) - Static method in class org.apache.beam.runners.spark.structuredstreaming.io.BoundedDatasetFactory: Create a Dataset for a BoundedSource via a Spark Table.
CreateStream<T> - Class in org.apache.beam.runners.spark.io: Create an input stream from Queue.
createTranslationContext(JavaSparkContext, SparkPipelineOptions, JobInfo) - Method in class org.apache.beam.runners.spark.translation.SparkBatchPortablePipelineTranslator
createTranslationContext(JavaSparkContext, SparkPipelineOptions, JobInfo) - Method in interface org.apache.beam.runners.spark.translation.SparkPortablePipelineTranslator
createTranslationContext(JavaSparkContext, SparkPipelineOptions, JobInfo) - Method in class org.apache.beam.runners.spark.translation.SparkStreamingPortablePipelineTranslator
CsvSink - Class in org.apache.beam.runners.spark.metrics.sink: A Sink for Spark's metric system reporting metrics (including Beam step metrics) to a CSV file.
CsvSink(Properties, MetricRegistry) - Constructor for class org.apache.beam.runners.spark.metrics.sink.CsvSink: Constructor for Spark 3.2.x and later.
CsvSink(Properties, MetricRegistry, SecurityManager) - Constructor for class org.apache.beam.runners.spark.metrics.sink.CsvSink: Constructor for Spark 3.1.x and earlier.
ctxt - Variable in class org.apache.beam.runners.spark.SparkRunner.Evaluator
currentInputWatermarkTime() - Method in class org.apache.beam.runners.spark.stateful.SparkTimerInternals
currentOutputWatermarkTime() - Method in class org.apache.beam.runners.spark.stateful.SparkTimerInternals
currentProcessingTime() - Method in class org.apache.beam.runners.spark.stateful.SparkTimerInternals
currentSynchronizedProcessingTime() - Method in class org.apache.beam.runners.spark.stateful.SparkTimerInternals

D

Dataset - Interface in org.apache.beam.runners.spark.translation: Holder for Spark RDD/DStream.
DEFAULT_MASTER_URL - Static variable in interface org.apache.beam.runners.spark.SparkCommonPipelineOptions
DefaultStopPipelineWatermarkFactory() - Constructor for class org.apache.beam.runners.spark.TestSparkPipelineOptions.DefaultStopPipelineWatermarkFactory
deleteTimer(StateNamespace, String, String) - Method in class org.apache.beam.runners.spark.stateful.SparkTimerInternals
deleteTimer(StateNamespace, String, String, TimeDomain) - Method in class org.apache.beam.runners.spark.stateful.SparkTimerInternals
deleteTimer(TimerInternals.TimerData) - Method in class org.apache.beam.runners.spark.stateful.SparkTimerInternals
deserializeOneOf(Expression, List<Encoder<T>>) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderHelpers
deserializeTimers(Collection<byte[]>, TimerInternals.TimerDataCoderV2) - Static method in class org.apache.beam.runners.spark.stateful.SparkTimerInternals
detectStreamingMode(Pipeline, StreamingOptions) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.PipelineTranslator: Analyse the pipeline to determine if we have to switch to streaming mode for the pipeline translation and update StreamingOptions accordingly.
dropExpiredTimers(SparkTimerInternals, WindowingStrategy<?, W>) - Static method in class org.apache.beam.runners.spark.util.TimerUtils
dStreamValues(JavaPairDStream<T1, T2>) - Static method in class org.apache.beam.runners.spark.translation.TranslationUtils: Transform a pair stream into a value stream.

E

empty() - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions.SparkSideInputReader
emptyBatch() - Method in class org.apache.beam.runners.spark.io.CreateStream: Adds an empty batch.
EmptyCheckpointMark - Class in org.apache.beam.runners.spark.io: Passing null values to Spark's Java API may cause problems because of Guava preconditions.
emptyList() - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.utils.ScalaInterop
EmptyListenersList() - Constructor for class org.apache.beam.runners.spark.SparkContextOptions.EmptyListenersList
emptyVoidFunction() - Static method in class org.apache.beam.runners.spark.translation.TranslationUtils
encoderFactory() - Method in interface org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderProvider
EncoderFactory - Class in org.apache.beam.runners.spark.structuredstreaming.translation.helpers
EncoderFactory() - Constructor for class org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderFactory
encoderFor(Coder<T>) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderHelpers: Creates a Spark Encoder for EncoderHelpers of BinaryType delegating to a Beam Coder underneath.
EncoderHelpers - Class in org.apache.beam.runners.spark.structuredstreaming.translation.helpers: Encoders utility class.
EncoderHelpers() - Constructor for class org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderHelpers
EncoderHelpers.Utils - Class in org.apache.beam.runners.spark.structuredstreaming.translation.helpers: Encoder / expression utils that are called from generated code.
encoderOf(Class<? super T>) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderHelpers: Gets or creates a default Encoder for EncoderHelpers.
encoderOf(Coder<T>) - Method in interface org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderProvider
encoderOf(Coder<T>, EncoderProvider.Factory<T>) - Method in interface org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderProvider
encoderOf(Coder<T>, EncoderProvider.Factory<T>) - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator.Context
EncoderProvider - Interface in org.apache.beam.runners.spark.structuredstreaming.translation.helpers
EncoderProvider.Factory<T> - Interface in org.apache.beam.runners.spark.structuredstreaming.translation.helpers
enterCompositeTransform(TransformHierarchy.Node) - Method in class org.apache.beam.runners.spark.SparkNativePipelineVisitor
enterCompositeTransform(TransformHierarchy.Node) - Method in class org.apache.beam.runners.spark.SparkRunner.Evaluator
equals(Object) - Method in class org.apache.beam.runners.spark.io.EmptyCheckpointMark
equals(Object) - Method in class org.apache.beam.runners.spark.io.MicrobatchSource
equals(Object) - Method in class org.apache.beam.runners.spark.util.ByteArray
evaluate() - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.EvaluationContext: Trigger evaluation of all leaf datasets.
evaluate(String, Dataset<T>) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.EvaluationContext: The purpose of this utility is to mark the evaluation of Spark actions, both during Pipeline translation, when evaluation is required, and when finally evaluating the pipeline.
evaluate(TransformT, EvaluationContext) - Method in interface org.apache.beam.runners.spark.translation.TransformEvaluator
EvaluationContext - Class in org.apache.beam.runners.spark.structuredstreaming.translation: The EvaluationContext is the result of a pipeline translation and can be used to evaluate / run the pipeline.
EvaluationContext - Class in org.apache.beam.runners.spark.translation: The EvaluationContext allows us to define pipeline instructions and translate between PObject<T>s or PCollection<T>s and Ts or DStreams/RDDs of Ts.
EvaluationContext(JavaSparkContext, Pipeline, PipelineOptions) - Constructor for class org.apache.beam.runners.spark.translation.EvaluationContext
EvaluationContext(JavaSparkContext, Pipeline, PipelineOptions, JavaStreamingContext) - Constructor for class org.apache.beam.runners.spark.translation.EvaluationContext
Evaluator(SparkPipelineTranslator, EvaluationContext) - Constructor for class org.apache.beam.runners.spark.SparkRunner.Evaluator
expand(PBegin) - Method in class org.apache.beam.runners.spark.io.CreateStream
expand(PCollection<String>) - Method in class org.apache.beam.runners.spark.examples.WordCount.CountWords
expand(PCollection<String>) - Method in class org.apache.beam.runners.spark.structuredstreaming.examples.WordCount.CountWords
expand(PCollection<T>) - Method in class org.apache.beam.runners.spark.io.ConsoleIO.Write.Unbound
EXPLODE_WINDOWS - org.apache.beam.runners.spark.translation.SparkCombineFn.WindowedAccumulator.Type
extractOutput() - Method in interface org.apache.beam.runners.spark.translation.SparkCombineFn.WindowedAccumulator: Extract output.
extractOutputStream(SparkCombineFn.WindowedAccumulator<?, ?, AccumT, ?>) - Method in class org.apache.beam.runners.spark.translation.SparkCombineFn: Extracts the stream of accumulated values.

F

finalizeCheckpoint() - Method in class org.apache.beam.runners.spark.io.EmptyCheckpointMark
FormatAsTextFn() - Constructor for class org.apache.beam.runners.spark.examples.WordCount.FormatAsTextFn
FormatAsTextFn() - Constructor for class org.apache.beam.runners.spark.structuredstreaming.examples.WordCount.FormatAsTextFn
forRegistry(MetricRegistry) - Static method in class org.apache.beam.runners.spark.metrics.WithMetricsSupport
forRegistry(MetricRegistry) - Static method in class org.apache.beam.runners.spark.structuredstreaming.metrics.WithMetricsSupport
forStreamFromSources(List<Integer>, Map<Integer, GlobalWatermarkHolder.SparkWatermarks>) - Static method in class org.apache.beam.runners.spark.stateful.SparkTimerInternals: Build the TimerInternals according to the feeding streams.
fromByteArray(byte[], Coder<T>) - Static method in class org.apache.beam.runners.spark.coders.CoderHelpers: Utility method for deserializing a byte array using the specified coder.
fromByteArray(byte[], Coder<T>) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.helpers.CoderHelpers: Utility method for deserializing a byte array using the specified coder.
fromByteArrays(Collection<byte[]>, Coder<T>) - Static method in class org.apache.beam.runners.spark.coders.CoderHelpers: Utility method for deserializing a Iterable of byte arrays using the specified coder.
fromByteFunction(Coder<T>) - Static method in class org.apache.beam.runners.spark.coders.CoderHelpers: A function wrapper for converting a byte array to an object.
FromByteFunction(Coder<K>, Coder<V>) - Constructor for class org.apache.beam.runners.spark.coders.CoderHelpers.FromByteFunction
fromByteFunctionIterable(Coder<K>, Coder<V>) - Static method in class org.apache.beam.runners.spark.coders.CoderHelpers: A function wrapper for converting a byte array pair to a key-value pair, where values are Iterable.
fromConfig(SparkJobServerDriver.SparkServerConfiguration) - Static method in class org.apache.beam.runners.spark.SparkJobServerDriver
fromOptions(PipelineOptions) - Static method in class org.apache.beam.runners.spark.SparkRunner: Creates and returns a new SparkRunner with specified options.
fromOptions(PipelineOptions) - Static method in class org.apache.beam.runners.spark.SparkRunnerDebugger
fromOptions(PipelineOptions) - Static method in class org.apache.beam.runners.spark.structuredstreaming.SparkStructuredStreamingRunner: Creates and returns a new SparkStructuredStreamingRunner with specified options.
fromOptions(PipelineOptions) - Static method in class org.apache.beam.runners.spark.TestSparkRunner
fromParams(String[]) - Static method in class org.apache.beam.runners.spark.SparkJobServerDriver
fun1(ScalaInterop.Fun1<T, V>) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.utils.ScalaInterop
fun2(ScalaInterop.Fun2<T1, T2, V>) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.utils.ScalaInterop
functionToFlatMapFunction(Function<InputT, OutputT>) - Static method in class org.apache.beam.runners.spark.translation.TranslationUtils: A utility method that adapts Function to a FlatMapFunction with an Iterator input.
fuse(PipelineTranslator.UnresolvedTranslation<T, T2>) - Method in interface org.apache.beam.runners.spark.structuredstreaming.translation.PipelineTranslator.UnresolvedTranslation

G

get() - Static method in class org.apache.beam.runners.spark.io.EmptyCheckpointMark
get(Long) - Static method in class org.apache.beam.runners.spark.util.GlobalWatermarkHolder: Returns the Broadcast containing the GlobalWatermarkHolder.SparkWatermarks mapped to their sources.
get(JobInfo) - Method in class org.apache.beam.runners.spark.translation.SparkExecutableStageContextFactory
get(BoundedWindow) - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions.SideInputValues.ByWindow
get(BoundedWindow) - Method in interface org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions.SideInputValues
get(BoundedWindow) - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions.SideInputValues.Global
get(PCollectionView<T>, BoundedWindow) - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions.CachedSideInputReader
get(PCollectionView<T>, BoundedWindow) - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions.SparkSideInputReader
get(PCollectionView<T>, BoundedWindow) - Method in class org.apache.beam.runners.spark.util.CachedSideInputReader
get(PCollectionView<T>, BoundedWindow) - Method in class org.apache.beam.runners.spark.util.SparkSideInputReader
get(PValue) - Method in class org.apache.beam.runners.spark.translation.EvaluationContext: Retrieve an object of Type T associated with the PValue passed in.
getBatchDuration() - Method in class org.apache.beam.runners.spark.io.CreateStream
getBatches() - Method in class org.apache.beam.runners.spark.io.CreateStream: Get the underlying queue representing the mock stream of micro-batches.
getBatchIntervalMillis() - Method in interface org.apache.beam.runners.spark.SparkPipelineOptions
getBeamCheckpointDir() - Method in class org.apache.beam.runners.spark.translation.streaming.Checkpoint.CheckpointDir
getBroadcastSizeEstimate() - Method in class org.apache.beam.runners.spark.util.SideInputBroadcast
getBundleSize() - Method in interface org.apache.beam.runners.spark.SparkPipelineOptions
getCacheCandidates() - Method in class org.apache.beam.runners.spark.translation.EvaluationContext: Get the map of cache candidates hold by the evaluation context.
getCheckpointDir() - Method in interface org.apache.beam.runners.spark.SparkCommonPipelineOptions
getCheckpointDurationMillis() - Method in interface org.apache.beam.runners.spark.SparkPipelineOptions
getCheckpointMark() - Method in class org.apache.beam.runners.spark.io.MicrobatchSource.Reader
getCheckpointMarkCoder() - Method in class org.apache.beam.runners.spark.io.MicrobatchSource
getCounters(MetricFilter) - Method in class org.apache.beam.runners.spark.metrics.WithMetricsSupport
getCounters(MetricFilter) - Method in class org.apache.beam.runners.spark.structuredstreaming.metrics.WithMetricsSupport
getCurrent() - Method in class org.apache.beam.runners.spark.io.MicrobatchSource.Reader
getCurrentSource() - Method in class org.apache.beam.runners.spark.io.MicrobatchSource.Reader
getCurrentTimestamp() - Method in class org.apache.beam.runners.spark.io.MicrobatchSource.Reader
getCurrentTransform() - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator.Context
getCurrentTransform() - Method in class org.apache.beam.runners.spark.translation.EvaluationContext
getDataset(PCollection<T>) - Method in interface org.apache.beam.runners.spark.structuredstreaming.translation.PipelineTranslator.TranslationState
getDataset(PCollection<T>) - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator.Context
getDefaultOverrides(boolean) - Static method in class org.apache.beam.runners.spark.SparkTransformOverrides
getDStream() - Method in class org.apache.beam.runners.spark.translation.streaming.UnboundedDataset
getEnableSparkMetricSinks() - Method in interface org.apache.beam.runners.spark.SparkCommonPipelineOptions
getExpectedAssertions() - Method in interface org.apache.beam.runners.spark.TestSparkPipelineOptions
getFirstTimestamp() - Method in class org.apache.beam.runners.spark.translation.SparkStreamingTranslationContext
getGauges(MetricFilter) - Method in class org.apache.beam.runners.spark.metrics.WithMetricsSupport
getGauges(MetricFilter) - Method in class org.apache.beam.runners.spark.structuredstreaming.metrics.WithMetricsSupport
getHighWatermark() - Method in class org.apache.beam.runners.spark.util.GlobalWatermarkHolder.SparkWatermarks
getHistograms(MetricFilter) - Method in class org.apache.beam.runners.spark.metrics.WithMetricsSupport
getHistograms(MetricFilter) - Method in class org.apache.beam.runners.spark.structuredstreaming.metrics.WithMetricsSupport
getId() - Method in class org.apache.beam.runners.spark.io.MicrobatchSource
getInput() - Method in interface org.apache.beam.runners.spark.structuredstreaming.translation.PipelineTranslator.UnresolvedTranslation
getInput() - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator.Context
getInput(PTransform<T, ?>) - Method in class org.apache.beam.runners.spark.translation.EvaluationContext
getInputFile() - Method in interface org.apache.beam.runners.spark.examples.WordCount.WordCountOptions
getInputFile() - Method in interface org.apache.beam.runners.spark.structuredstreaming.examples.WordCount.WordCountOptions
getInputs() - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator.Context
getInputs(PTransform<?, ?>) - Method in class org.apache.beam.runners.spark.translation.EvaluationContext
getInstance() - Static method in class org.apache.beam.runners.spark.metrics.MetricsAccumulator
getInstance() - Static method in class org.apache.beam.runners.spark.translation.SparkExecutableStageContextFactory
getInstance(SparkSession) - Static method in class org.apache.beam.runners.spark.structuredstreaming.metrics.MetricsAccumulator: Get the MetricsAccumulator on this driver.
getLastWatermarkedBatchTime() - Static method in class org.apache.beam.runners.spark.util.GlobalWatermarkHolder
getListeners() - Method in interface org.apache.beam.runners.spark.SparkContextOptions
getLowWatermark() - Method in class org.apache.beam.runners.spark.util.GlobalWatermarkHolder.SparkWatermarks
getMaxRecordsPerBatch() - Method in interface org.apache.beam.runners.spark.SparkPipelineOptions
getMeters(MetricFilter) - Method in class org.apache.beam.runners.spark.metrics.WithMetricsSupport
getMeters(MetricFilter) - Method in class org.apache.beam.runners.spark.structuredstreaming.metrics.WithMetricsSupport
getMinReadTimeMillis() - Method in interface org.apache.beam.runners.spark.SparkPipelineOptions
getNum() - Method in class org.apache.beam.runners.spark.io.ConsoleIO.Write.Unbound
getOptions() - Method in interface org.apache.beam.runners.spark.structuredstreaming.translation.PipelineTranslator.TranslationState
getOptions() - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator.Context
getOptions() - Method in class org.apache.beam.runners.spark.translation.EvaluationContext
getOptionsSupplier() - Method in interface org.apache.beam.runners.spark.structuredstreaming.translation.PipelineTranslator.TranslationState
getOptionsSupplier() - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator.Context
getOrCreateReader(PipelineOptions, CheckpointMarkT) - Method in class org.apache.beam.runners.spark.io.MicrobatchSource
getOrCreateSession(SparkStructuredStreamingPipelineOptions) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.SparkSessionFactory: Gets active SparkSession or creates one using SparkStructuredStreamingPipelineOptions.
getOrDecode(Coder<T>) - Method in class org.apache.beam.runners.spark.translation.ValueAndCoderLazySerializable
getOutput() - Method in interface org.apache.beam.runners.spark.examples.WordCount.WordCountOptions
getOutput() - Method in interface org.apache.beam.runners.spark.structuredstreaming.examples.WordCount.WordCountOptions
getOutput() - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator.Context
getOutput(PTransform<?, T>) - Method in class org.apache.beam.runners.spark.translation.EvaluationContext
getOutput(TupleTag<T>) - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator.Context
getOutputCoder() - Method in class org.apache.beam.runners.spark.io.MicrobatchSource
getOutputCoders() - Method in class org.apache.beam.runners.spark.translation.EvaluationContext
getOutputs() - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator.Context
getOutputs(PTransform<?, ?>) - Method in class org.apache.beam.runners.spark.translation.EvaluationContext
getPartitions() - Method in class org.apache.beam.runners.spark.io.SourceRDD.Bounded
getPartitions() - Method in class org.apache.beam.runners.spark.io.SourceRDD.Unbounded
getPipeline() - Method in class org.apache.beam.runners.spark.translation.EvaluationContext
getPipelineOptions() - Method in class org.apache.beam.runners.spark.SparkRunnerRegistrar.Options
getPipelineOptions() - Method in class org.apache.beam.runners.spark.structuredstreaming.SparkStructuredStreamingRunnerRegistrar.Options
getPipelineRunners() - Method in class org.apache.beam.runners.spark.SparkRunnerRegistrar.Runner
getPipelineRunners() - Method in class org.apache.beam.runners.spark.structuredstreaming.SparkStructuredStreamingRunnerRegistrar.Runner
getProvidedSparkContext() - Method in interface org.apache.beam.runners.spark.SparkContextOptions
getPViews() - Method in class org.apache.beam.runners.spark.translation.EvaluationContext: Return the current views creates in the pipeline.
getRDD() - Method in class org.apache.beam.runners.spark.translation.BoundedDataset
getReadTimePercentage() - Method in interface org.apache.beam.runners.spark.SparkPipelineOptions
getRootCheckpointDir() - Method in class org.apache.beam.runners.spark.translation.streaming.Checkpoint.CheckpointDir
getSerializableOptions() - Method in class org.apache.beam.runners.spark.translation.EvaluationContext
getSerializableOptions() - Method in class org.apache.beam.runners.spark.translation.SparkTranslationContext
getSideInputBroadcast(PCollection<T>, SideInputValues.Loader<T>) - Method in interface org.apache.beam.runners.spark.structuredstreaming.translation.PipelineTranslator.TranslationState
getSideInputBroadcast(PCollection<T>, SideInputValues.Loader<T>) - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator.Context
getSideInputs(Iterable<PCollectionView<?>>, JavaSparkContext, SparkPCollectionView) - Static method in class org.apache.beam.runners.spark.translation.TranslationUtils: Create SideInputs as Broadcast variables.
getSparkCheckpointDir() - Method in class org.apache.beam.runners.spark.translation.streaming.Checkpoint.CheckpointDir
getSparkContext() - Method in class org.apache.beam.runners.spark.translation.EvaluationContext
getSparkContext() - Method in class org.apache.beam.runners.spark.translation.SparkTranslationContext
getSparkContext(SparkPipelineOptions) - Static method in class org.apache.beam.runners.spark.translation.SparkContextFactory
getSparkMaster() - Method in interface org.apache.beam.runners.spark.SparkCommonPipelineOptions
getSparkSession() - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.EvaluationContext
getSparkSession() - Method in interface org.apache.beam.runners.spark.structuredstreaming.translation.PipelineTranslator.TranslationState
getSparkSession() - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator.Context
getState() - Method in class org.apache.beam.runners.spark.SparkPipelineResult
getState() - Method in class org.apache.beam.runners.spark.structuredstreaming.SparkStructuredStreamingPipelineResult
getStopPipelineWatermark() - Method in interface org.apache.beam.runners.spark.TestSparkPipelineOptions
getStorageLevel() - Method in interface org.apache.beam.runners.spark.SparkCommonPipelineOptions
getStreamingContext() - Method in class org.apache.beam.runners.spark.translation.EvaluationContext
getStreamingContext() - Method in class org.apache.beam.runners.spark.translation.SparkStreamingTranslationContext
getStreamingTimeoutMs() - Method in interface org.apache.beam.runners.spark.SparkPortableStreamingPipelineOptions
getStreamSources() - Method in class org.apache.beam.runners.spark.translation.streaming.UnboundedDataset
getSynchronizedProcessingTime() - Method in class org.apache.beam.runners.spark.util.GlobalWatermarkHolder.SparkWatermarks
getTestMode() - Method in interface org.apache.beam.runners.spark.structuredstreaming.SparkStructuredStreamingPipelineOptions: Set to true to run the job in test mode.
getTimers() - Method in class org.apache.beam.runners.spark.stateful.SparkTimerInternals
getTimers(MetricFilter) - Method in class org.apache.beam.runners.spark.metrics.WithMetricsSupport
getTimers(MetricFilter) - Method in class org.apache.beam.runners.spark.structuredstreaming.metrics.WithMetricsSupport
getTimes() - Method in class org.apache.beam.runners.spark.io.CreateStream: Get times so they can be pushed into the GlobalWatermarkHolder.
getTransformPayloadTranslators() - Method in class org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator.SparkTransformsRegistrar
getTransformTranslator(TransformT) - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.batch.PipelineTranslatorBatch: Returns a TransformTranslator for the given PTransform if known.
getTransformTranslator(TransformT) - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.PipelineTranslator: Returns a TransformTranslator for the given PTransform if known.
getTupleTagCoders(Map<TupleTag<?>, PCollection<?>>) - Static method in class org.apache.beam.runners.spark.translation.TranslationUtils: Utility to get mapping between TupleTag and a coder.
getTupleTagDecodeFunction(Map<TupleTag<?>, Coder<WindowedValue<?>>>) - Static method in class org.apache.beam.runners.spark.translation.TranslationUtils: Returns a pair function to convert bytes to value via coder.
getTupleTagEncodeFunction(Map<TupleTag<?>, Coder<WindowedValue<?>>>) - Static method in class org.apache.beam.runners.spark.translation.TranslationUtils: Returns a pair function to convert value to bytes via coder.
getUseActiveSparkSession() - Method in interface org.apache.beam.runners.spark.structuredstreaming.SparkStructuredStreamingPipelineOptions
getUsesProvidedSparkContext() - Method in interface org.apache.beam.runners.spark.SparkPipelineOptions
getValue() - Method in class org.apache.beam.runners.spark.util.ByteArray
getValue() - Method in class org.apache.beam.runners.spark.util.SideInputBroadcast
getWatermark() - Method in class org.apache.beam.runners.spark.io.MicrobatchSource.Reader
global(Map<Integer, GlobalWatermarkHolder.SparkWatermarks>) - Static method in class org.apache.beam.runners.spark.stateful.SparkTimerInternals: Build a global TimerInternals for all feeding streams.
globally(CombineWithContext.CombineFnWithContext<InputT, AccumT, OutputT>, SerializablePipelineOptions, Map<TupleTag<?>, KV<WindowingStrategy<?, ?>, SideInputBroadcast<?>>>, WindowingStrategy<?, ?>) - Static method in class org.apache.beam.runners.spark.translation.SparkCombineFn
GlobalWatermarkHolder - Class in org.apache.beam.runners.spark.util: A store to hold the global watermarks for a micro-batch.
GlobalWatermarkHolder() - Constructor for class org.apache.beam.runners.spark.util.GlobalWatermarkHolder
GlobalWatermarkHolder.SparkWatermarks - Class in org.apache.beam.runners.spark.util: A GlobalWatermarkHolder.SparkWatermarks holds the watermarks and batch time relevant to a micro-batch input from a specific source.
GlobalWatermarkHolder.WatermarkAdvancingStreamingListener - Class in org.apache.beam.runners.spark.util: Advance the WMs onBatchCompleted event.
GraphiteSink - Class in org.apache.beam.runners.spark.metrics.sink: A Sink for Spark's metric system reporting metrics (including Beam step metrics) to Graphite.
GraphiteSink(Properties, MetricRegistry) - Constructor for class org.apache.beam.runners.spark.metrics.sink.GraphiteSink: Constructor for Spark 3.2.x and later.
GraphiteSink(Properties, MetricRegistry, SecurityManager) - Constructor for class org.apache.beam.runners.spark.metrics.sink.GraphiteSink: Constructor for Spark 3.1.x and earlier.
GroupAlsoByWindowViaOutputBufferFn<K,InputT,W extends org.apache.beam.sdk.transforms.windowing.BoundedWindow> - Class in org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions: A FlatMap function that groups by windows in batch mode using ReduceFnRunner.
GroupAlsoByWindowViaOutputBufferFn(WindowingStrategy<?, W>, StateInternalsFactory<K>, SystemReduceFn<K, InputT, Iterable<InputT>, Iterable<InputT>, W>, Supplier<PipelineOptions>) - Constructor for class org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions.GroupAlsoByWindowViaOutputBufferFn
groupByKeyAndWindow(JavaDStream<WindowedValue<KV<K, InputT>>>, Coder<K>, Coder<WindowedValue<InputT>>, WindowingStrategy<?, W>, SerializablePipelineOptions, List<Integer>, String) - Static method in class org.apache.beam.runners.spark.stateful.SparkGroupAlsoByWindowViaWindowSet
groupByKeyOnly(JavaRDD<WindowedValue<KV<K, V>>>, Coder<K>, WindowedValue.WindowedValueCoder<V>, Partitioner) - Static method in class org.apache.beam.runners.spark.translation.GroupCombineFunctions: An implementation of GroupByKeyViaGroupByKeyOnly.GroupByKeyOnly for the Spark runner.
GroupCombineFunctions - Class in org.apache.beam.runners.spark.translation: A set of group/combine functions to apply to Spark RDDs.
GroupCombineFunctions() - Constructor for class org.apache.beam.runners.spark.translation.GroupCombineFunctions
GroupNonMergingWindowsFunctions - Class in org.apache.beam.runners.spark.translation: Functions for GroupByKey with Non-Merging windows translations to Spark.
GroupNonMergingWindowsFunctions() - Constructor for class org.apache.beam.runners.spark.translation.GroupNonMergingWindowsFunctions

H

hashCode() - Method in class org.apache.beam.runners.spark.io.EmptyCheckpointMark
hashCode() - Method in class org.apache.beam.runners.spark.io.MicrobatchSource
hashCode() - Method in class org.apache.beam.runners.spark.util.ByteArray
hasTranslation(PTransform<?, ?>) - Method in interface org.apache.beam.runners.spark.translation.SparkPipelineTranslator
hasTranslation(PTransform<?, ?>) - Method in class org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator.Translator
hasTranslation(PTransform<?, ?>) - Method in class org.apache.beam.runners.spark.translation.TransformTranslator.Translator

I

init(SparkPipelineOptions, JavaSparkContext) - Static method in class org.apache.beam.runners.spark.metrics.MetricsAccumulator: Init metrics accumulator if it has not been initiated.
initAccumulators(SparkPipelineOptions, JavaSparkContext) - Static method in class org.apache.beam.runners.spark.SparkRunner: Init Metrics/Aggregators accumulators.
initialSystemTimeAt(Instant) - Method in class org.apache.beam.runners.spark.io.CreateStream: Set the initial synchronized processing time.
INSTANCE - Static variable in interface org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderProvider.Factory
invokeWithExecutor(RunnerApi.Pipeline, Struct, String, ListeningExecutorService) - Method in class org.apache.beam.runners.spark.SparkJobInvoker
isBoundedCollection(Collection<PCollection<?>>) - Method in class org.apache.beam.runners.spark.SparkRunner.Evaluator
isCacheDisabled() - Method in interface org.apache.beam.runners.spark.SparkPipelineOptions
isEmpty() - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions.CachedSideInputReader
isEmpty() - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions.SparkSideInputReader
isEmpty() - Method in interface org.apache.beam.runners.spark.translation.SparkCombineFn.WindowedAccumulator: Check if this accumulator is empty.
isEmpty() - Method in class org.apache.beam.runners.spark.util.CachedSideInputReader
isEmpty() - Method in class org.apache.beam.runners.spark.util.SparkSideInputReader
isForceStreaming() - Method in interface org.apache.beam.runners.spark.TestSparkPipelineOptions
isForceWatermarkSync() - Method in class org.apache.beam.runners.spark.io.CreateStream
isLeaf(PCollection<?>) - Method in interface org.apache.beam.runners.spark.structuredstreaming.translation.PipelineTranslator.TranslationState
isLeaf(PCollection<?>) - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator.Context
IsSparkNativeTransform() - Constructor for class org.apache.beam.runners.spark.translation.SparkBatchPortablePipelineTranslator.IsSparkNativeTransform
isZero() - Method in class org.apache.beam.runners.spark.metrics.MetricsContainerStepMapAccumulator
isZero() - Method in class org.apache.beam.runners.spark.structuredstreaming.metrics.MetricsAccumulator

J

javaIterator(Iterator<T>) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.utils.ScalaInterop: Java Iterator of Scala Iterator.

K

keyed(CombineWithContext.CombineFnWithContext<V, AccumT, OutputT>, SerializablePipelineOptions, Map<TupleTag<?>, KV<WindowingStrategy<?, ?>, SideInputBroadcast<?>>>, WindowingStrategy<?, ?>) - Static method in class org.apache.beam.runners.spark.translation.SparkCombineFn
keyEncoderOf(KvCoder<K, V>) - Method in interface org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderProvider
knownUrns() - Method in class org.apache.beam.runners.spark.translation.SparkBatchPortablePipelineTranslator
knownUrns() - Method in interface org.apache.beam.runners.spark.translation.SparkPortablePipelineTranslator
knownUrns() - Method in class org.apache.beam.runners.spark.translation.SparkStreamingPortablePipelineTranslator
kvEncoder(Encoder<K>, Encoder<V>) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderHelpers: Creates a Spark Encoder for KV of StructType with fields key and value.
kvEncoderOf(KvCoder<K, V>) - Method in interface org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderProvider

L

listOf(T) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.utils.ScalaInterop
loader(PCollection<T>) - Static method in interface org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions.SideInputValues: Factory to load SideInputValues from a Dataset based on the window strategy.

M

main(String[]) - Static method in class org.apache.beam.runners.spark.examples.WordCount
main(String[]) - Static method in class org.apache.beam.runners.spark.SparkJobServerDriver
main(String[]) - Static method in class org.apache.beam.runners.spark.SparkPipelineRunner: Main method to be called only as the entry point to an executable jar with structure as defined in PortablePipelineJarUtils.
main(String[]) - Static method in class org.apache.beam.runners.spark.structuredstreaming.examples.WordCount
mapEncoder(Encoder<K>, Encoder<V>, Class<MapT>) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderHelpers: Creates a Spark Encoder of MapType that deserializes to EncoderHelpers.
mapSourceFunction(SerializablePipelineOptions, String) - Static method in class org.apache.beam.runners.spark.stateful.StateSpecFunctions: A StateSpec function to support reading from an UnboundedSource.
match(Class<V>) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.utils.ScalaInterop
maxTimestamp(Iterable<BoundedWindow>) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderHelpers.Utils: The end of the only window (max timestamp).
merge(ImplT, SparkCombineFn<?, ?, AccumT, ?>) - Method in interface org.apache.beam.runners.spark.translation.SparkCombineFn.WindowedAccumulator: Merge other accumulator into this one.
merge(AccumulatorV2<MetricsContainerStepMap, MetricsContainerStepMap>) - Method in class org.apache.beam.runners.spark.metrics.MetricsContainerStepMapAccumulator
merge(AccumulatorV2<MetricsContainerStepMap, MetricsContainerStepMap>) - Method in class org.apache.beam.runners.spark.structuredstreaming.metrics.MetricsAccumulator
MERGING - org.apache.beam.runners.spark.translation.SparkCombineFn.WindowedAccumulator.Type
Metadata(long, Instant, Instant, long, MetricsContainerStepMap) - Constructor for class org.apache.beam.runners.spark.io.SparkUnboundedSource.Metadata
metricRegistry() - Method in class org.apache.beam.runners.spark.metrics.SparkBeamMetricSource
metricRegistry() - Method in class org.apache.beam.runners.spark.structuredstreaming.metrics.SparkBeamMetricSource
metrics() - Method in class org.apache.beam.runners.spark.SparkPipelineResult
metrics() - Method in class org.apache.beam.runners.spark.structuredstreaming.SparkStructuredStreamingPipelineResult
MetricsAccumulator - Class in org.apache.beam.runners.spark.metrics: For resilience, Accumulators are required to be wrapped in a Singleton.
MetricsAccumulator - Class in org.apache.beam.runners.spark.structuredstreaming.metrics: AccumulatorV2 for Beam metrics captured in MetricsContainerStepMap.
MetricsAccumulator() - Constructor for class org.apache.beam.runners.spark.metrics.MetricsAccumulator
MetricsAccumulator() - Constructor for class org.apache.beam.runners.spark.structuredstreaming.metrics.MetricsAccumulator
MetricsAccumulator.AccumulatorCheckpointingSparkListener - Class in org.apache.beam.runners.spark.metrics: Spark Listener which checkpoints MetricsContainerStepMap values for fault-tolerance.
MetricsContainerStepMapAccumulator - Class in org.apache.beam.runners.spark.metrics: AccumulatorV2 implementation for MetricsContainerStepMap.
MetricsContainerStepMapAccumulator(MetricsContainerStepMap) - Constructor for class org.apache.beam.runners.spark.metrics.MetricsContainerStepMapAccumulator
MicrobatchSource<T,CheckpointMarkT extends org.apache.beam.sdk.io.UnboundedSource.CheckpointMark> - Class in org.apache.beam.runners.spark.io: A Source that accommodates Spark's micro-batch oriented nature and wraps an UnboundedSource.
MicrobatchSource.Reader - Class in org.apache.beam.runners.spark.io: Mostly based on BoundedReadFromUnboundedSource's UnboundedToBoundedSourceAdapter, with some adjustments for Spark specifics.
MultiDoFnFunction<InputT,OutputT> - Class in org.apache.beam.runners.spark.translation: DoFunctions ignore outputs that are not the main output.
MultiDoFnFunction(MetricsContainerStepMapAccumulator, String, DoFn<InputT, OutputT>, SerializablePipelineOptions, TupleTag<OutputT>, List<TupleTag<?>>, Coder<InputT>, Map<TupleTag<?>, Coder<?>>, Map<TupleTag<?>, KV<WindowingStrategy<?, ?>, SideInputBroadcast<?>>>, WindowingStrategy<?, ?>, boolean, DoFnSchemaInformation, Map<String, PCollectionView<?>>, boolean) - Constructor for class org.apache.beam.runners.spark.translation.MultiDoFnFunction
mutablePairEncoder(Encoder<T1>, Encoder<T2>) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderHelpers: Creates a Spark Encoder for Spark's MutablePair of StructType with fields `_1` and `_2`.

N

nextBatch(TimestampedValue<T>...) - Method in class org.apache.beam.runners.spark.io.CreateStream: Enqueue next micro-batch elements.
nextBatch(T...) - Method in class org.apache.beam.runners.spark.io.CreateStream: For non-timestamped elements.
nextSinkId() - Method in class org.apache.beam.runners.spark.translation.SparkTranslationContext: Generate a unique pCollection id number to identify runner-generated sinks.
NON_MERGING - org.apache.beam.runners.spark.translation.SparkCombineFn.WindowedAccumulator.Type
NoOpStepContext - Class in org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions: A StepContext for Spark Batch Runner execution.
NoOpStepContext() - Constructor for class org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions.NoOpStepContext

O

of(SideInputReader) - Static method in class org.apache.beam.runners.spark.util.CachedSideInputReader: Create a new cached SideInputReader.
of(SideInputReader, Collection<PCollectionView<?>>) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions.CachedSideInputReader: Creates a SideInputReader that caches results for costly Materializations if present, otherwise the SideInputReader is returned as is.
of(Coder<T>, Duration) - Static method in class org.apache.beam.runners.spark.io.CreateStream: Creates a new Spark based stream without forced watermark sync, intended for test purposes.
of(Coder<T>, Duration, boolean) - Static method in class org.apache.beam.runners.spark.io.CreateStream: Creates a new Spark based stream intended for test purposes.
of(T, Coder<T>) - Static method in class org.apache.beam.runners.spark.translation.ValueAndCoderLazySerializable
onBatchCompleted(JavaStreamingListenerBatchCompleted) - Method in class org.apache.beam.runners.spark.metrics.MetricsAccumulator.AccumulatorCheckpointingSparkListener
onBatchCompleted(JavaStreamingListenerBatchCompleted) - Method in class org.apache.beam.runners.spark.util.GlobalWatermarkHolder.WatermarkAdvancingStreamingListener
oneOfEncoder(List<Encoder<T>>) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderHelpers: Creates a one-of Spark Encoder of StructType where each alternative is represented as colum / field named by its index with a separate Encoder each.
Options() - Constructor for class org.apache.beam.runners.spark.SparkRunnerRegistrar.Options
Options() - Constructor for class org.apache.beam.runners.spark.structuredstreaming.SparkStructuredStreamingRunnerRegistrar.Options
org.apache.beam.runners.spark - package org.apache.beam.runners.spark: Internal implementation of the Beam runner for Apache Spark.
org.apache.beam.runners.spark.coders - package org.apache.beam.runners.spark.coders: Beam coders and coder-related utilities for running on Apache Spark.
org.apache.beam.runners.spark.examples - package org.apache.beam.runners.spark.examples
org.apache.beam.runners.spark.io - package org.apache.beam.runners.spark.io: Spark-specific transforms for I/O.
org.apache.beam.runners.spark.metrics - package org.apache.beam.runners.spark.metrics: Provides internal utilities for implementing Beam metrics using Spark accumulators.
org.apache.beam.runners.spark.metrics.sink - package org.apache.beam.runners.spark.metrics.sink: Spark sinks that supports beam metrics and aggregators.
org.apache.beam.runners.spark.stateful - package org.apache.beam.runners.spark.stateful: Spark-specific stateful operators.
org.apache.beam.runners.spark.structuredstreaming - package org.apache.beam.runners.spark.structuredstreaming: Internal implementation of the Beam runner for Apache Spark.
org.apache.beam.runners.spark.structuredstreaming.examples - package org.apache.beam.runners.spark.structuredstreaming.examples
org.apache.beam.runners.spark.structuredstreaming.io - package org.apache.beam.runners.spark.structuredstreaming.io: Spark-specific transforms for I/O.
org.apache.beam.runners.spark.structuredstreaming.metrics - package org.apache.beam.runners.spark.structuredstreaming.metrics: Provides internal utilities for implementing Beam metrics using Spark accumulators.
org.apache.beam.runners.spark.structuredstreaming.metrics.sink - package org.apache.beam.runners.spark.structuredstreaming.metrics.sink: Spark sinks that supports beam metrics and aggregators.
org.apache.beam.runners.spark.structuredstreaming.translation - package org.apache.beam.runners.spark.structuredstreaming.translation: Internal translators for running Beam pipelines on Spark.
org.apache.beam.runners.spark.structuredstreaming.translation.batch - package org.apache.beam.runners.spark.structuredstreaming.translation.batch: Internal utilities to translate Beam pipelines to Spark batching.
org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions - package org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions: Internal implementation of the Beam runner for Apache Spark.
org.apache.beam.runners.spark.structuredstreaming.translation.helpers - package org.apache.beam.runners.spark.structuredstreaming.translation.helpers: Internal helpers to translate Beam pipelines to Spark streaming.
org.apache.beam.runners.spark.structuredstreaming.translation.utils - package org.apache.beam.runners.spark.structuredstreaming.translation.utils: Internal utils to translate Beam pipelines to Spark streaming.
org.apache.beam.runners.spark.translation - package org.apache.beam.runners.spark.translation: Internal translators for running Beam pipelines on Spark.
org.apache.beam.runners.spark.translation.streaming - package org.apache.beam.runners.spark.translation.streaming: Internal utilities to translate Beam pipelines to Spark streaming.
org.apache.beam.runners.spark.util - package org.apache.beam.runners.spark.util: Internal utilities to translate Beam pipelines to Spark.
out() - Static method in class org.apache.beam.runners.spark.io.ConsoleIO.Write: Prints 10 elements from the PCollection to the console.
out(int) - Static method in class org.apache.beam.runners.spark.io.ConsoleIO.Write: Prints num elements from the PCollection to stdout.

P

pairFunctionToPairFlatMapFunction(PairFunction<T, K, V>) - Static method in class org.apache.beam.runners.spark.translation.TranslationUtils: A utility method that adapts PairFunction to a PairFlatMapFunction with an Iterator input.
paneInfoFromBytes(byte[]) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderHelpers.Utils
paneInfoToBytes(PaneInfo) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderHelpers.Utils
partitioner() - Method in class org.apache.beam.runners.spark.io.SourceRDD.Unbounded
PipelineTranslator - Class in org.apache.beam.runners.spark.structuredstreaming.translation: The pipeline translator translates a Beam Pipeline into a Spark correspondence, that can then be evaluated.
PipelineTranslator() - Constructor for class org.apache.beam.runners.spark.structuredstreaming.translation.PipelineTranslator
PipelineTranslator.TranslationState - Interface in org.apache.beam.runners.spark.structuredstreaming.translation: Shared, mutable state during the translation of a pipeline and omitted afterwards.
PipelineTranslator.UnresolvedTranslation<InT,T> - Interface in org.apache.beam.runners.spark.structuredstreaming.translation: Unresolved translation, allowing to optimize the generated Spark DAG.
PipelineTranslatorBatch - Class in org.apache.beam.runners.spark.structuredstreaming.translation.batch: PipelineTranslator for executing a Pipeline in Spark in batch mode.
PipelineTranslatorBatch() - Constructor for class org.apache.beam.runners.spark.structuredstreaming.translation.batch.PipelineTranslatorBatch
popDataset(String) - Method in class org.apache.beam.runners.spark.translation.SparkTranslationContext: Retrieve the dataset for the pCollection id and remove it from the DAG's leaves.
prepareFilesToStage(SparkCommonPipelineOptions) - Static method in interface org.apache.beam.runners.spark.SparkCommonPipelineOptions: Classpath contains non jar files (eg.
pushDataset(String, Dataset) - Method in class org.apache.beam.runners.spark.translation.SparkTranslationContext: Add output of transform to context.
putDataset(PTransform<?, ? extends PValue>, Dataset) - Method in class org.apache.beam.runners.spark.translation.EvaluationContext: Add single output of transform to context map and possibly cache if it conforms EvaluationContext.shouldCache(PTransform, PValue).
putDataset(PCollection<T>, Dataset<WindowedValue<T>>) - Method in interface org.apache.beam.runners.spark.structuredstreaming.translation.PipelineTranslator.TranslationState
putDataset(PCollection<T>, Dataset<WindowedValue<T>>, boolean) - Method in interface org.apache.beam.runners.spark.structuredstreaming.translation.PipelineTranslator.TranslationState
putDataset(PCollection<T>, Dataset<WindowedValue<T>>, boolean) - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator.Context
putDataset(PValue, Dataset) - Method in class org.apache.beam.runners.spark.translation.EvaluationContext: Add output of transform to context map and possibly cache if it conforms EvaluationContext.shouldCache(PTransform, PValue).
putPView(PCollectionView<?>, Iterable<WindowedValue<?>>, Coder<Iterable<WindowedValue<?>>>) - Method in class org.apache.beam.runners.spark.translation.EvaluationContext: Adds/Replaces a view to the current views creates in the pipeline.
putUnresolved(PCollection<OutT>, PipelineTranslator.UnresolvedTranslation<InT, OutT>) - Method in interface org.apache.beam.runners.spark.structuredstreaming.translation.PipelineTranslator.TranslationState
putUnresolved(PCollection<T>, PipelineTranslator.UnresolvedTranslation<InputT, T>) - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator.Context

R

read(Kryo, Input) - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions.SideInputValues.BaseSideInputValues
read(Kryo, Input, Class<ValueAndCoderLazySerializable<T>>) - Method in class org.apache.beam.runners.spark.translation.ValueAndCoderKryoSerializer
read(JavaStreamingContext, SerializablePipelineOptions, UnboundedSource<T, CheckpointMarkT>, String) - Static method in class org.apache.beam.runners.spark.io.SparkUnboundedSource
readObject(FileSystem, Path) - Static method in class org.apache.beam.runners.spark.translation.streaming.Checkpoint
registerClasses(Kryo) - Method in class org.apache.beam.runners.spark.coders.SparkRunnerKryoRegistrator
registerClasses(Kryo) - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.SparkSessionFactory.SparkKryoRegistrator
ReifyTimestampsAndWindowsFunction<K,V> - Class in org.apache.beam.runners.spark.translation: Simple Function to bring the windowing information into the value from the implicit background representation of the PCollection.
ReifyTimestampsAndWindowsFunction() - Constructor for class org.apache.beam.runners.spark.translation.ReifyTimestampsAndWindowsFunction
rejectStateAndTimers(DoFn<?, ?>) - Static method in class org.apache.beam.runners.spark.translation.TranslationUtils: Reject state and timers DoFn.
replace(Class<V>, T) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.utils.ScalaInterop
replaceTransforms(Pipeline, StreamingOptions) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.PipelineTranslator
report() - Method in class org.apache.beam.runners.spark.metrics.sink.CsvSink
report() - Method in class org.apache.beam.runners.spark.metrics.sink.GraphiteSink
report() - Method in class org.apache.beam.runners.spark.structuredstreaming.metrics.sink.CodahaleCsvSink
report() - Method in class org.apache.beam.runners.spark.structuredstreaming.metrics.sink.CodahaleGraphiteSink
reset() - Method in class org.apache.beam.runners.spark.metrics.MetricsContainerStepMapAccumulator
reset() - Method in class org.apache.beam.runners.spark.structuredstreaming.metrics.MetricsAccumulator
reshuffle(JavaRDD<WindowedValue<T>>, WindowedValue.WindowedValueCoder<T>) - Static method in class org.apache.beam.runners.spark.translation.GroupCombineFunctions: An implementation of Reshuffle for the Spark runner.
resolve(Supplier<PipelineOptions>, Dataset<WindowedValue<InT>>) - Method in interface org.apache.beam.runners.spark.structuredstreaming.translation.PipelineTranslator.UnresolvedTranslation
run(RunnerApi.Pipeline, JobInfo) - Method in class org.apache.beam.runners.spark.SparkPipelineRunner
run(Pipeline) - Method in class org.apache.beam.runners.spark.SparkRunner
run(Pipeline) - Method in class org.apache.beam.runners.spark.SparkRunnerDebugger
run(Pipeline) - Method in class org.apache.beam.runners.spark.structuredstreaming.SparkStructuredStreamingRunner
run(Pipeline) - Method in class org.apache.beam.runners.spark.TestSparkRunner
Runner() - Constructor for class org.apache.beam.runners.spark.SparkRunnerRegistrar.Runner
Runner() - Constructor for class org.apache.beam.runners.spark.structuredstreaming.SparkStructuredStreamingRunnerRegistrar.Runner

S

ScalaInterop - Class in org.apache.beam.runners.spark.structuredstreaming.translation.utils: Utilities for easier interoperability with the Spark Scala API.
ScalaInterop.Fun1<T,V> - Interface in org.apache.beam.runners.spark.structuredstreaming.translation.utils
ScalaInterop.Fun2<T1,T2,V> - Interface in org.apache.beam.runners.spark.structuredstreaming.translation.utils
scalaIterator(Iterable<T>) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.utils.ScalaInterop: Scala Iterator of Java Iterable.
scalaIterator(Iterator<T>) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.utils.ScalaInterop: Scala Iterator of Java Iterator.
seqOf(T...) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.utils.ScalaInterop
serializeOneOf(Expression, List<Encoder<T>>) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderHelpers
serializeTimers(Collection<TimerInternals.TimerData>, TimerInternals.TimerDataCoderV2) - Static method in class org.apache.beam.runners.spark.stateful.SparkTimerInternals
sessionBuilder(String) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.SparkSessionFactory: Creates Spark session builder with some optimizations for local mode, e.g.
setBatchIntervalMillis(Long) - Method in interface org.apache.beam.runners.spark.SparkPipelineOptions
setBundleSize(Long) - Method in interface org.apache.beam.runners.spark.SparkPipelineOptions
setCacheDisabled(boolean) - Method in interface org.apache.beam.runners.spark.SparkPipelineOptions
setCheckpointDir(String) - Method in interface org.apache.beam.runners.spark.SparkCommonPipelineOptions
setCheckpointDurationMillis(Long) - Method in interface org.apache.beam.runners.spark.SparkPipelineOptions
setCurrentTransform(AppliedPTransform<?, ?, ?>) - Method in class org.apache.beam.runners.spark.translation.EvaluationContext
setEnableSparkMetricSinks(Boolean) - Method in interface org.apache.beam.runners.spark.SparkCommonPipelineOptions
setExpectedAssertions(Integer) - Method in interface org.apache.beam.runners.spark.TestSparkPipelineOptions
setForceStreaming(boolean) - Method in interface org.apache.beam.runners.spark.TestSparkPipelineOptions
setInputFile(String) - Method in interface org.apache.beam.runners.spark.examples.WordCount.WordCountOptions
setInputFile(String) - Method in interface org.apache.beam.runners.spark.structuredstreaming.examples.WordCount.WordCountOptions
setListeners(List<JavaStreamingListener>) - Method in interface org.apache.beam.runners.spark.SparkContextOptions
setMaxRecordsPerBatch(Long) - Method in interface org.apache.beam.runners.spark.SparkPipelineOptions
setMinReadTimeMillis(Long) - Method in interface org.apache.beam.runners.spark.SparkPipelineOptions
setName(String) - Method in class org.apache.beam.runners.spark.translation.BoundedDataset
setName(String) - Method in interface org.apache.beam.runners.spark.translation.Dataset
setName(String) - Method in class org.apache.beam.runners.spark.translation.streaming.UnboundedDataset
setOutput(String) - Method in interface org.apache.beam.runners.spark.examples.WordCount.WordCountOptions
setOutput(String) - Method in interface org.apache.beam.runners.spark.structuredstreaming.examples.WordCount.WordCountOptions
setProvidedSparkContext(JavaSparkContext) - Method in interface org.apache.beam.runners.spark.SparkContextOptions
setProvidedSparkContext(JavaSparkContext) - Static method in class org.apache.beam.runners.spark.translation.SparkContextFactory: Set an externally managed JavaSparkContext that will be used if SparkPipelineOptions.getUsesProvidedSparkContext() is set to true.
setReadTimePercentage(Double) - Method in interface org.apache.beam.runners.spark.SparkPipelineOptions
setSparkMaster(String) - Method in interface org.apache.beam.runners.spark.SparkCommonPipelineOptions
setStopPipelineWatermark(Long) - Method in interface org.apache.beam.runners.spark.TestSparkPipelineOptions
setStorageLevel(String) - Method in interface org.apache.beam.runners.spark.SparkCommonPipelineOptions
setStreamingTimeoutMs(Long) - Method in interface org.apache.beam.runners.spark.SparkPortableStreamingPipelineOptions
setTestMode(boolean) - Method in interface org.apache.beam.runners.spark.structuredstreaming.SparkStructuredStreamingPipelineOptions
setTimer(StateNamespace, String, String, Instant, Instant, TimeDomain) - Method in class org.apache.beam.runners.spark.stateful.SparkTimerInternals
setTimer(TimerInternals.TimerData) - Method in class org.apache.beam.runners.spark.stateful.SparkTimerInternals
setUseActiveSparkSession(boolean) - Method in interface org.apache.beam.runners.spark.structuredstreaming.SparkStructuredStreamingPipelineOptions
setUsesProvidedSparkContext(boolean) - Method in interface org.apache.beam.runners.spark.SparkPipelineOptions
shouldCache(PTransform<?, ? extends PValue>, PValue) - Method in class org.apache.beam.runners.spark.translation.EvaluationContext: Cache PCollection if SparkPipelineOptions.isCacheDisabled is false or transform isn't GroupByKey transformation and PCollection is used more then once in Pipeline.
shouldDefer(TransformHierarchy.Node) - Method in class org.apache.beam.runners.spark.SparkRunner.Evaluator
SideInputBroadcast<T> - Class in org.apache.beam.runners.spark.util: Broadcast helper for side inputs.
SideInputValues<T> - Interface in org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions: SideInputValues serves as a Kryo serializable container that contains a materialized view of side inputs.
SideInputValues.BaseSideInputValues<BinaryT,ValuesT extends @NonNull java.lang.Object,T> - Class in org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions
SideInputValues.ByWindow<T> - Class in org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions: General SideInputValues for BoundedWindows in two possible states.
SideInputValues.Global<T> - Class in org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions: Specialized SideInputValues for use with the GlobalWindow in two possible states.
SideInputValues.Loader<T> - Interface in org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions: Factory function for load SideInputValues from a Dataset.
SINGLE_WINDOW - org.apache.beam.runners.spark.translation.SparkCombineFn.WindowedAccumulator.Type
skipAssignWindows(Window.Assign<T>, EvaluationContext) - Static method in class org.apache.beam.runners.spark.translation.TranslationUtils: Checks if the window transformation should be applied or skipped.
sourceName() - Method in class org.apache.beam.runners.spark.metrics.SparkBeamMetricSource
sourceName() - Method in class org.apache.beam.runners.spark.structuredstreaming.metrics.SparkBeamMetricSource
SourceRDD - Class in org.apache.beam.runners.spark.io: Classes implementing Beam Source RDDs.
SourceRDD() - Constructor for class org.apache.beam.runners.spark.io.SourceRDD
SourceRDD.Bounded<T> - Class in org.apache.beam.runners.spark.io: A SourceRDD.Bounded reads input from a BoundedSource and creates a Spark RDD.
SourceRDD.Unbounded<T,CheckpointMarkT extends org.apache.beam.sdk.io.UnboundedSource.CheckpointMark> - Class in org.apache.beam.runners.spark.io: A SourceRDD.Unbounded is the implementation of a micro-batch in a SourceDStream.
SparkAssignWindowFn<T,W extends org.apache.beam.sdk.transforms.windowing.BoundedWindow> - Class in org.apache.beam.runners.spark.translation: An implementation of Window.Assign for the Spark runner.
SparkAssignWindowFn(WindowFn<? super T, W>) - Constructor for class org.apache.beam.runners.spark.translation.SparkAssignWindowFn
SparkBatchPortablePipelineTranslator - Class in org.apache.beam.runners.spark.translation: Translates a bounded portable pipeline into a Spark job.
SparkBatchPortablePipelineTranslator() - Constructor for class org.apache.beam.runners.spark.translation.SparkBatchPortablePipelineTranslator
SparkBatchPortablePipelineTranslator.IsSparkNativeTransform - Class in org.apache.beam.runners.spark.translation: Predicate to determine whether a URN is a Spark native transform.
SparkBeamMetricSource - Class in org.apache.beam.runners.spark.metrics: A Spark Source that is tailored to expose a SparkBeamMetric, wrapping an underlying MetricResults instance.
SparkBeamMetricSource - Class in org.apache.beam.runners.spark.structuredstreaming.metrics: A Spark Source that is tailored to expose a SparkBeamMetric, wrapping an underlying MetricResults instance.
SparkBeamMetricSource(String) - Constructor for class org.apache.beam.runners.spark.metrics.SparkBeamMetricSource
SparkBeamMetricSource(String, MetricsAccumulator) - Constructor for class org.apache.beam.runners.spark.structuredstreaming.metrics.SparkBeamMetricSource
SparkCombineFn<InputT,ValueT,AccumT,OutputT> - Class in org.apache.beam.runners.spark.translation: A CombineFnBase.GlobalCombineFn with a CombineWithContext.Context for the SparkRunner.
SparkCombineFn.WindowedAccumulator<InputT,ValueT,AccumT,ImplT extends SparkCombineFn.WindowedAccumulator<InputT,ValueT,AccumT,ImplT>> - Interface in org.apache.beam.runners.spark.translation: Accumulator of WindowedValues holding values for different windows.
SparkCombineFn.WindowedAccumulator.Type - Enum in org.apache.beam.runners.spark.translation: Type of the accumulator.
SparkCommonPipelineOptions - Interface in org.apache.beam.runners.spark: Spark runner PipelineOptions handles Spark execution-related configurations, such as the master address, and other user-related knobs.
SparkCommonPipelineOptions.StorageLevelFactory - Class in org.apache.beam.runners.spark: Returns Spark's default storage level for the Dataset or RDD API based on the respective runner.
SparkCommonPipelineOptions.TmpCheckpointDirFactory - Class in org.apache.beam.runners.spark: Returns the default checkpoint directory of /tmp/${job.name}.
SparkContextFactory - Class in org.apache.beam.runners.spark.translation
SparkContextOptions - Interface in org.apache.beam.runners.spark: A custom PipelineOptions to work with properties related to JavaSparkContext.
SparkContextOptions.EmptyListenersList - Class in org.apache.beam.runners.spark: Returns an empty list, to avoid handling null.
SparkExecutableStageContextFactory - Class in org.apache.beam.runners.spark.translation: Singleton class that contains one ExecutableStageContext.Factory per job.
SparkGroupAlsoByWindowViaWindowSet - Class in org.apache.beam.runners.spark.stateful: An implementation of GroupByKeyViaGroupByKeyOnly.GroupAlsoByWindow logic for grouping by windows and controlling trigger firings and pane accumulation.
SparkGroupAlsoByWindowViaWindowSet() - Constructor for class org.apache.beam.runners.spark.stateful.SparkGroupAlsoByWindowViaWindowSet
SparkGroupAlsoByWindowViaWindowSet.StateAndTimers - Class in org.apache.beam.runners.spark.stateful: State and Timers wrapper.
SparkJobInvoker - Class in org.apache.beam.runners.spark: Creates a job invocation to manage the Spark runner's execution of a portable pipeline.
SparkJobServerDriver - Class in org.apache.beam.runners.spark: Driver program that starts a job server for the Spark runner.
SparkJobServerDriver.SparkServerConfiguration - Class in org.apache.beam.runners.spark: Spark runner-specific Configuration for the jobServer.
SparkKryoRegistrator() - Constructor for class org.apache.beam.runners.spark.structuredstreaming.translation.SparkSessionFactory.SparkKryoRegistrator
SparkNativePipelineVisitor - Class in org.apache.beam.runners.spark: Pipeline visitor for translating a Beam pipeline into equivalent Spark operations.
SparkPCollectionView - Class in org.apache.beam.runners.spark.translation: SparkPCollectionView is used to pass serialized views to lambdas.
SparkPCollectionView() - Constructor for class org.apache.beam.runners.spark.translation.SparkPCollectionView
SparkPipelineOptions - Interface in org.apache.beam.runners.spark: Spark runner PipelineOptions handles Spark execution-related configurations, such as the master address, batch-interval, and other user-related knobs.
SparkPipelineResult - Class in org.apache.beam.runners.spark: Represents a Spark pipeline execution result.
SparkPipelineRunner - Class in org.apache.beam.runners.spark: Runs a portable pipeline on Apache Spark.
SparkPipelineRunner(SparkPipelineOptions) - Constructor for class org.apache.beam.runners.spark.SparkPipelineRunner
SparkPipelineTranslator - Interface in org.apache.beam.runners.spark.translation: Translator to support translation between Beam transformations and Spark transformations.
SparkPortablePipelineTranslator<T extends SparkTranslationContext> - Interface in org.apache.beam.runners.spark.translation: Interface for portable Spark translators.
SparkPortableStreamingPipelineOptions - Interface in org.apache.beam.runners.spark: Pipeline options specific to the Spark portable runner running a streaming job.
SparkRunner - Class in org.apache.beam.runners.spark: The SparkRunner translate operations defined on a pipeline to a representation executable by Spark, and then submitting the job to Spark to be executed.
SparkRunner.Evaluator - Class in org.apache.beam.runners.spark: Evaluator on the pipeline.
SparkRunnerDebugger - Class in org.apache.beam.runners.spark: Pipeline runner which translates a Beam pipeline into equivalent Spark operations, without running them.
SparkRunnerDebugger.DebugSparkPipelineResult - Class in org.apache.beam.runners.spark: PipelineResult of running a Pipeline using SparkRunnerDebugger Use SparkRunnerDebugger.DebugSparkPipelineResult.getDebugString() to get a String representation of the Pipeline translated into Spark native operations.
SparkRunnerKryoRegistrator - Class in org.apache.beam.runners.spark.coders: Custom KryoRegistrators for Beam's Spark runner needs and registering used class in spark translation for better serialization performance.
SparkRunnerKryoRegistrator() - Constructor for class org.apache.beam.runners.spark.coders.SparkRunnerKryoRegistrator
SparkRunnerRegistrar - Class in org.apache.beam.runners.spark: Contains the PipelineRunnerRegistrar and PipelineOptionsRegistrar for the SparkRunner.
SparkRunnerRegistrar.Options - Class in org.apache.beam.runners.spark: Registers the SparkPipelineOptions.
SparkRunnerRegistrar.Runner - Class in org.apache.beam.runners.spark: Registers the SparkRunner.
SparkRunnerStreamingContextFactory - Class in org.apache.beam.runners.spark.translation.streaming: A JavaStreamingContext factory for resilience.
SparkRunnerStreamingContextFactory(Pipeline, SparkPipelineOptions, Checkpoint.CheckpointDir) - Constructor for class org.apache.beam.runners.spark.translation.streaming.SparkRunnerStreamingContextFactory
SparkServerConfiguration() - Constructor for class org.apache.beam.runners.spark.SparkJobServerDriver.SparkServerConfiguration
SparkSessionFactory - Class in org.apache.beam.runners.spark.structuredstreaming.translation
SparkSessionFactory() - Constructor for class org.apache.beam.runners.spark.structuredstreaming.translation.SparkSessionFactory
SparkSessionFactory.SparkKryoRegistrator - Class in org.apache.beam.runners.spark.structuredstreaming.translation: KryoRegistrator for Spark to serialize broadcast variables used for side-inputs.
SparkSideInputReader - Class in org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions: SideInputReader using broadcasted SideInputValues.
SparkSideInputReader - Class in org.apache.beam.runners.spark.util: A SideInputReader for the SparkRunner.
SparkSideInputReader(Map<TupleTag<?>, KV<WindowingStrategy<?, ?>, SideInputBroadcast<?>>>) - Constructor for class org.apache.beam.runners.spark.util.SparkSideInputReader
SparkStreamingPortablePipelineTranslator - Class in org.apache.beam.runners.spark.translation: Translates an unbounded portable pipeline into a Spark job.
SparkStreamingPortablePipelineTranslator() - Constructor for class org.apache.beam.runners.spark.translation.SparkStreamingPortablePipelineTranslator
SparkStreamingTranslationContext - Class in org.apache.beam.runners.spark.translation: Translation context used to lazily store Spark datasets during streaming portable pipeline translation and compute them after translation.
SparkStreamingTranslationContext(JavaSparkContext, SparkPipelineOptions, JobInfo) - Constructor for class org.apache.beam.runners.spark.translation.SparkStreamingTranslationContext
SparkStructuredStreamingPipelineOptions - Interface in org.apache.beam.runners.spark.structuredstreaming: Spark runner PipelineOptions handles Spark execution-related configurations, such as the master address, and other user-related knobs.
SparkStructuredStreamingPipelineResult - Class in org.apache.beam.runners.spark.structuredstreaming
SparkStructuredStreamingRunner - Class in org.apache.beam.runners.spark.structuredstreaming: A Spark runner build on top of Spark's SQL Engine (Structured Streaming framework).
SparkStructuredStreamingRunnerRegistrar - Class in org.apache.beam.runners.spark.structuredstreaming: Contains the PipelineRunnerRegistrar and PipelineOptionsRegistrar for the SparkStructuredStreamingRunner.
SparkStructuredStreamingRunnerRegistrar.Options - Class in org.apache.beam.runners.spark.structuredstreaming: Registers the SparkStructuredStreamingPipelineOptions.
SparkStructuredStreamingRunnerRegistrar.Runner - Class in org.apache.beam.runners.spark.structuredstreaming: Registers the SparkStructuredStreamingRunner.
SparkTimerInternals - Class in org.apache.beam.runners.spark.stateful: An implementation of TimerInternals for the SparkRunner.
SparkTransformOverrides - Class in org.apache.beam.runners.spark: PTransform overrides for Spark runner.
SparkTransformOverrides() - Constructor for class org.apache.beam.runners.spark.SparkTransformOverrides
SparkTransformsRegistrar() - Constructor for class org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator.SparkTransformsRegistrar
SparkTranslationContext - Class in org.apache.beam.runners.spark.translation: Translation context used to lazily store Spark data sets during portable pipeline translation and compute them after translation.
SparkTranslationContext(JavaSparkContext, PipelineOptions, JobInfo) - Constructor for class org.apache.beam.runners.spark.translation.SparkTranslationContext
SparkUnboundedSource - Class in org.apache.beam.runners.spark.io: A "composite" InputDStream implementation for UnboundedSources.
SparkUnboundedSource() - Constructor for class org.apache.beam.runners.spark.io.SparkUnboundedSource
SparkUnboundedSource.Metadata - Class in org.apache.beam.runners.spark.io: A metadata holder for an input stream partition.
SparkWatermarks(Instant, Instant, Instant) - Constructor for class org.apache.beam.runners.spark.util.GlobalWatermarkHolder.SparkWatermarks
start() - Method in class org.apache.beam.runners.spark.io.MicrobatchSource.Reader
start() - Method in class org.apache.beam.runners.spark.metrics.sink.CsvSink
start() - Method in class org.apache.beam.runners.spark.metrics.sink.GraphiteSink
start() - Method in class org.apache.beam.runners.spark.structuredstreaming.metrics.sink.CodahaleCsvSink
start() - Method in class org.apache.beam.runners.spark.structuredstreaming.metrics.sink.CodahaleGraphiteSink
start() - Method in class org.apache.beam.runners.spark.translation.streaming.TestDStream
stateInternals() - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions.NoOpStepContext
StateSpecFunctions - Class in org.apache.beam.runners.spark.stateful: A class containing StateSpec mappingFunctions.
StateSpecFunctions() - Constructor for class org.apache.beam.runners.spark.stateful.StateSpecFunctions
stop() - Method in class org.apache.beam.runners.spark.metrics.sink.CsvSink
stop() - Method in class org.apache.beam.runners.spark.metrics.sink.GraphiteSink
stop() - Method in class org.apache.beam.runners.spark.SparkPipelineResult
stop() - Method in class org.apache.beam.runners.spark.SparkRunnerDebugger.DebugSparkPipelineResult
stop() - Method in class org.apache.beam.runners.spark.structuredstreaming.metrics.sink.CodahaleCsvSink
stop() - Method in class org.apache.beam.runners.spark.structuredstreaming.metrics.sink.CodahaleGraphiteSink
stop() - Method in class org.apache.beam.runners.spark.translation.streaming.TestDStream
stopSparkContext(JavaSparkContext) - Static method in class org.apache.beam.runners.spark.translation.SparkContextFactory
storageLevel() - Method in class org.apache.beam.runners.spark.translation.EvaluationContext
StorageLevelFactory() - Constructor for class org.apache.beam.runners.spark.SparkCommonPipelineOptions.StorageLevelFactory
StreamingTransformTranslator - Class in org.apache.beam.runners.spark.translation.streaming: Supports translation between a Beam transform, and Spark's operations on DStreams.
StreamingTransformTranslator.SparkTransformsRegistrar - Class in org.apache.beam.runners.spark.translation.streaming: Registers classes specialized by the Spark runner.
StreamingTransformTranslator.Translator - Class in org.apache.beam.runners.spark.translation.streaming: Translator matches Beam transformation with the appropriate evaluator.

T

test(RunnerApi.PTransform) - Method in class org.apache.beam.runners.spark.translation.SparkBatchPortablePipelineTranslator.IsSparkNativeTransform
TEST_REUSE_SPARK_CONTEXT - Static variable in class org.apache.beam.runners.spark.translation.SparkContextFactory: Deprecated.
This will leak your SparkContext, any attempt to create a new SparkContext later will fail. Please use SparkContextFactory.setProvidedSparkContext(JavaSparkContext) / SparkContextFactory.clearProvidedSparkContext() instead to properly control the lifecycle of your context. Alternatively you may also provide a SparkContext using SparkPipelineOptions.setUsesProvidedSparkContext(boolean) together with SparkContextOptions.setProvidedSparkContext(JavaSparkContext) and close that one appropriately. Tests of this module should use SparkContextRule.
TestDStream<T> - Class in org.apache.beam.runners.spark.translation.streaming
TestDStream(TestStream<T>, StreamingContext) - Constructor for class org.apache.beam.runners.spark.translation.streaming.TestDStream
TestSparkPipelineOptions - Interface in org.apache.beam.runners.spark: A SparkPipelineOptions for tests.
TestSparkPipelineOptions.DefaultStopPipelineWatermarkFactory - Class in org.apache.beam.runners.spark: A factory to provide the default watermark to stop a pipeline that reads from an unbounded source.
TestSparkRunner - Class in org.apache.beam.runners.spark: The SparkRunner translate operations defined on a pipeline to a representation executable by Spark, and then submitting the job to Spark to be executed.
timerInternals() - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions.NoOpStepContext
TimerUtils - Class in org.apache.beam.runners.spark.util
TimerUtils() - Constructor for class org.apache.beam.runners.spark.util.TimerUtils
TmpCheckpointDirFactory() - Constructor for class org.apache.beam.runners.spark.SparkCommonPipelineOptions.TmpCheckpointDirFactory
toByteArray(T, Coder<T>) - Static method in class org.apache.beam.runners.spark.coders.CoderHelpers: Utility method for serializing an object using the specified coder.
toByteArray(T, Coder<T>) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.helpers.CoderHelpers: Utility method for serializing an object using the specified coder.
toByteArrays(Iterable<T>, Coder<T>) - Static method in class org.apache.beam.runners.spark.coders.CoderHelpers: Utility method for serializing a Iterable of values using the specified coder.
toByteArrayWithTs(T, Coder<T>, Instant) - Static method in class org.apache.beam.runners.spark.coders.CoderHelpers: Utility method for serializing an object using the specified coder, appending timestamp representation.
toByteFunction(Coder<K>, Coder<V>) - Static method in class org.apache.beam.runners.spark.coders.CoderHelpers: A function wrapper for converting a key-value pair to a byte array pair.
toByteFunction(Coder<T>) - Static method in class org.apache.beam.runners.spark.coders.CoderHelpers: A function wrapper for converting an object to a bytearray.
toByteFunctionWithTs(Coder<K>, Coder<V>, Function<Tuple2<K, V>, Instant>) - Static method in class org.apache.beam.runners.spark.coders.CoderHelpers: A function wrapper for converting a key-value pair to a byte array pair, where the key in resulting ByteArray contains (key, timestamp).
toMap(ArrayData, ArrayData, DataType, DataType) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderHelpers.Utils
toNativeString() - Method in interface org.apache.beam.runners.spark.translation.TransformEvaluator
toPairByKeyInWindowedValue(Coder<K>) - Static method in class org.apache.beam.runners.spark.translation.TranslationUtils: Extract key from a WindowedValue KV into a pair.
toPairFlatMapFunction() - Static method in class org.apache.beam.runners.spark.translation.TranslationUtils: KV to pair flatmap function.
toPairFunction() - Static method in class org.apache.beam.runners.spark.translation.TranslationUtils: KV to pair function.
toSeq(Collection<Object>) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderHelpers.Utils
toSeq(ArrayData) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderHelpers.Utils
toString() - Method in class org.apache.beam.runners.spark.stateful.SparkTimerInternals
toString() - Method in class org.apache.beam.runners.spark.util.GlobalWatermarkHolder.SparkWatermarks
toTreeMap(ArrayData, ArrayData, DataType, DataType) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderHelpers.Utils
TRANSFORM_URN - Static variable in class org.apache.beam.runners.spark.io.ConsoleIO.Write.Unbound
TRANSFORM_URN - Static variable in class org.apache.beam.runners.spark.io.CreateStream
TransformEvaluator<TransformT extends org.apache.beam.sdk.transforms.PTransform<?,?>> - Interface in org.apache.beam.runners.spark.translation: Describe a PTransform evaluator.
TransformTranslator<InT extends org.apache.beam.sdk.values.PInput,OutT extends org.apache.beam.sdk.values.POutput,TransformT extends org.apache.beam.sdk.transforms.PTransform<InT,OutT>> - Class in org.apache.beam.runners.spark.structuredstreaming.translation: A TransformTranslator provides the capability to translate a specific primitive or composite PTransform into its Spark correspondence.
TransformTranslator - Class in org.apache.beam.runners.spark.translation: Supports translation between a Beam transform, and Spark's operations on RDDs.
TransformTranslator(float) - Constructor for class org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator
TransformTranslator.Context - Class in org.apache.beam.runners.spark.structuredstreaming.translation: Available mutable context to translate a PTransform.
TransformTranslator.Translator - Class in org.apache.beam.runners.spark.translation: Translator matches Beam transformation with the appropriate evaluator.
translate(RunnerApi.Pipeline, SparkStreamingTranslationContext) - Method in class org.apache.beam.runners.spark.translation.SparkStreamingPortablePipelineTranslator: Translates pipeline from Beam into the Spark context.
translate(RunnerApi.Pipeline, SparkTranslationContext) - Method in class org.apache.beam.runners.spark.translation.SparkBatchPortablePipelineTranslator: Translates pipeline from Beam into the Spark context.
translate(RunnerApi.Pipeline, T) - Method in interface org.apache.beam.runners.spark.translation.SparkPortablePipelineTranslator: Translates the given pipeline.
translate(Pipeline, SparkSession, SparkCommonPipelineOptions) - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.PipelineTranslator: Translates a Beam pipeline into its Spark correspondence using the Spark SQL / Dataset API.
translate(TransformHierarchy.Node, TransformT) - Method in class org.apache.beam.runners.spark.SparkRunner.Evaluator: Determine if this Node belongs to a Bounded branch of the pipeline, or Unbounded, and translate with the proper translator.
translate(TransformT, TransformTranslator.Context) - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator
translateBounded(PTransform<?, ?>) - Method in interface org.apache.beam.runners.spark.translation.SparkPipelineTranslator
translateBounded(PTransform<?, ?>) - Method in class org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator.Translator
translateBounded(PTransform<?, ?>) - Method in class org.apache.beam.runners.spark.translation.TransformTranslator.Translator
translateUnbounded(PTransform<?, ?>) - Method in interface org.apache.beam.runners.spark.translation.SparkPipelineTranslator
translateUnbounded(PTransform<?, ?>) - Method in class org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator.Translator
translateUnbounded(PTransform<?, ?>) - Method in class org.apache.beam.runners.spark.translation.TransformTranslator.Translator
TranslationUtils - Class in org.apache.beam.runners.spark.translation: A set of utilities to help translating Beam transformations into Spark transformations.
TranslationUtils.CombineGroupedValues<K,InputT,OutputT> - Class in org.apache.beam.runners.spark.translation: A SparkCombineFn function applied to grouped KVs.
TranslationUtils.TupleTagFilter<V> - Class in org.apache.beam.runners.spark.translation: A utility class to filter TupleTags.
translator - Variable in class org.apache.beam.runners.spark.SparkRunner.Evaluator
Translator() - Constructor for class org.apache.beam.runners.spark.translation.TransformTranslator.Translator
Translator(SparkPipelineTranslator) - Constructor for class org.apache.beam.runners.spark.translation.streaming.StreamingTransformTranslator.Translator
tuple(T1, T2) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.utils.ScalaInterop
tupleEncoder(Encoder<T1>, Encoder<T2>) - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator.Context
TupleTagFilter(TupleTag<V>) - Constructor for class org.apache.beam.runners.spark.translation.TranslationUtils.TupleTagFilter

U

Unbounded(SparkContext, SerializablePipelineOptions, MicrobatchSource<T, CheckpointMarkT>, int) - Constructor for class org.apache.beam.runners.spark.io.SourceRDD.Unbounded
UnboundedDataset<T> - Class in org.apache.beam.runners.spark.translation.streaming: DStream holder Can also crate a DStream from a supplied queue of values, but mainly for testing.
UnboundedDataset(JavaDStream<WindowedValue<T>>, List<Integer>) - Constructor for class org.apache.beam.runners.spark.translation.streaming.UnboundedDataset
unpersist() - Method in class org.apache.beam.runners.spark.util.SideInputBroadcast
updateCacheCandidates(Pipeline, SparkPipelineTranslator, EvaluationContext) - Static method in class org.apache.beam.runners.spark.SparkRunner: Evaluator that update/populate the cache candidates.
Utils() - Constructor for class org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderHelpers.Utils

V

validate() - Method in class org.apache.beam.runners.spark.io.MicrobatchSource
validateMaterializations(Iterable<PCollectionView<?>>) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions.SparkSideInputReader
value() - Method in class org.apache.beam.runners.spark.metrics.MetricsContainerStepMapAccumulator
value() - Method in class org.apache.beam.runners.spark.structuredstreaming.metrics.MetricsAccumulator
ValueAndCoderKryoSerializer<T> - Class in org.apache.beam.runners.spark.translation: Kryo serializer for ValueAndCoderLazySerializable.
ValueAndCoderKryoSerializer() - Constructor for class org.apache.beam.runners.spark.translation.ValueAndCoderKryoSerializer
ValueAndCoderLazySerializable<T> - Class in org.apache.beam.runners.spark.translation: A holder object that lets you serialize an element with a Coder with minimal wasted space.
valueEncoderOf(KvCoder<K, V>) - Method in interface org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderProvider
valueOf(String) - Static method in enum org.apache.beam.runners.spark.translation.SparkCombineFn.WindowedAccumulator.Type: Returns the enum constant of this type with the specified name.
values() - Static method in enum org.apache.beam.runners.spark.translation.SparkCombineFn.WindowedAccumulator.Type: Returns an array containing the constants of this enum type, in the order they are declared.
visitPrimitiveTransform(TransformHierarchy.Node) - Method in class org.apache.beam.runners.spark.SparkRunner.Evaluator

W

waitUntilFinish() - Method in class org.apache.beam.runners.spark.SparkPipelineResult
waitUntilFinish() - Method in class org.apache.beam.runners.spark.structuredstreaming.SparkStructuredStreamingPipelineResult
waitUntilFinish(Duration) - Method in class org.apache.beam.runners.spark.SparkPipelineResult
waitUntilFinish(Duration) - Method in class org.apache.beam.runners.spark.structuredstreaming.SparkStructuredStreamingPipelineResult
WatermarkAdvancingStreamingListener() - Constructor for class org.apache.beam.runners.spark.util.GlobalWatermarkHolder.WatermarkAdvancingStreamingListener
windowCoder(PCollection<T>) - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator
windowedEncoder(Coder<T>) - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator.Context
windowedEncoder(Coder<T>, Coder<W>) - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator.Context
windowedEncoder(Encoder<T>) - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator.Context
windowedValueEncoder(Encoder<T>, Encoder<W>) - Static method in class org.apache.beam.runners.spark.structuredstreaming.translation.helpers.EncoderHelpers: Creates a Spark Encoder for EncoderHelpers of StructType with fields value, timestamp, windows and pane.
windowEncoder() - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.TransformTranslator.Context
WithMetricsSupport - Class in org.apache.beam.runners.spark.metrics: A MetricRegistry decorator-like that supports AggregatorMetric and SparkBeamMetric as Gauges.
WithMetricsSupport - Class in org.apache.beam.runners.spark.structuredstreaming.metrics: A MetricRegistry decorator-like that supports BeamMetricSets as Gauges.
WordCount - Class in org.apache.beam.runners.spark.examples: Duplicated from beam-examples-java to avoid dependency.
WordCount - Class in org.apache.beam.runners.spark.structuredstreaming.examples: Duplicated from beam-examples-java to avoid dependency.
WordCount() - Constructor for class org.apache.beam.runners.spark.examples.WordCount
WordCount() - Constructor for class org.apache.beam.runners.spark.structuredstreaming.examples.WordCount
WordCount.CountWords - Class in org.apache.beam.runners.spark.examples: A PTransform that converts a PCollection containing lines of text into a PCollection of formatted word counts.
WordCount.CountWords - Class in org.apache.beam.runners.spark.structuredstreaming.examples: A PTransform that converts a PCollection containing lines of text into a PCollection of formatted word counts.
WordCount.FormatAsTextFn - Class in org.apache.beam.runners.spark.examples: A SimpleFunction that converts a Word and Count into a printable string.
WordCount.FormatAsTextFn - Class in org.apache.beam.runners.spark.structuredstreaming.examples: A SimpleFunction that converts a Word and Count into a printable string.
WordCount.WordCountOptions - Interface in org.apache.beam.runners.spark.examples: Options supported by WordCount.
WordCount.WordCountOptions - Interface in org.apache.beam.runners.spark.structuredstreaming.examples: Options supported by WordCount.
write(Kryo, Output) - Method in class org.apache.beam.runners.spark.structuredstreaming.translation.batch.functions.SideInputValues.BaseSideInputValues
write(Kryo, Output, ValueAndCoderLazySerializable<T>) - Method in class org.apache.beam.runners.spark.translation.ValueAndCoderKryoSerializer
writeObject(FileSystem, Path, Object) - Static method in class org.apache.beam.runners.spark.translation.streaming.Checkpoint

A B C D E F G H I J K L M N O P R S T U V W
All Classes All Packages