Package org.apache.beam.runners.spark

Interface SparkContextOptions

All Superinterfaces:

org.apache.beam.sdk.options.ApplicationNameOptions, org.apache.beam.sdk.options.FileStagingOptions, org.apache.beam.sdk.transforms.display.HasDisplayData, org.apache.beam.sdk.options.PipelineOptions, SparkCommonPipelineOptions, SparkPipelineOptions, org.apache.beam.sdk.options.StreamingOptions

public interface SparkContextOptions
extends SparkPipelineOptions

A custom PipelineOptions to work with properties related to JavaSparkContext.

This can only be used programmatically (as opposed to passing command line arguments), since the properties here are context-aware and should not be propagated to workers.

Separating this from SparkPipelineOptions is needed so the context-aware properties, which link to Spark dependencies, won't be scanned by PipelineOptions reflective instantiation. Note that SparkContextOptions is not registered with SparkRunnerRegistrar.

Note: It's recommended to use SparkContextFactory.setProvidedSparkContext(JavaSparkContext) instead of setProvidedSparkContext(JavaSparkContext) for testing. When using @TestPipeline any provided JavaSparkContext via SparkContextOptions is dropped.

Nested Class Summary

Nested Classes

Modifier and Type	Interface	Description
static class	SparkContextOptions.EmptyListenersList	Returns an empty list, to avoid handling null.

Nested classes/interfaces inherited from interface org.apache.beam.sdk.options.PipelineOptions

org.apache.beam.sdk.options.PipelineOptions.AtomicLongFactory, org.apache.beam.sdk.options.PipelineOptions.CheckEnabled, org.apache.beam.sdk.options.PipelineOptions.DirectRunner, org.apache.beam.sdk.options.PipelineOptions.JobNameFactory, org.apache.beam.sdk.options.PipelineOptions.UserAgentFactory

Nested classes/interfaces inherited from interface org.apache.beam.runners.spark.SparkCommonPipelineOptions

SparkCommonPipelineOptions.StorageLevelFactory, SparkCommonPipelineOptions.TmpCheckpointDirFactory

Field Summary

Fields inherited from interface org.apache.beam.runners.spark.SparkCommonPipelineOptions

DEFAULT_MASTER_URL

Method Summary

Modifier and Type	Method	Description
java.util.List<org.apache.spark.streaming.api.java.JavaStreamingListener>	getListeners()
org.apache.spark.api.java.JavaSparkContext	getProvidedSparkContext()
void	setListeners(java.util.List<org.apache.spark.streaming.api.java.JavaStreamingListener> listeners)
void	setProvidedSparkContext(org.apache.spark.api.java.JavaSparkContext jsc)

Methods inherited from interface org.apache.beam.sdk.options.ApplicationNameOptions

getAppName, setAppName

Methods inherited from interface org.apache.beam.sdk.options.FileStagingOptions

getFilesToStage, setFilesToStage

Methods inherited from interface org.apache.beam.sdk.transforms.display.HasDisplayData

populateDisplayData

Methods inherited from interface org.apache.beam.sdk.options.PipelineOptions

as, getJobName, getOptionsId, getRunner, getStableUniqueNames, getTempLocation, getUserAgent, outputRuntimeOptions, revision, setJobName, setOptionsId, setRunner, setStableUniqueNames, setTempLocation, setUserAgent

Methods inherited from interface org.apache.beam.runners.spark.SparkCommonPipelineOptions

getCheckpointDir, getEnableSparkMetricSinks, getSparkMaster, getStorageLevel, setCheckpointDir, setEnableSparkMetricSinks, setSparkMaster, setStorageLevel

Methods inherited from interface org.apache.beam.runners.spark.SparkPipelineOptions

getBatchIntervalMillis, getBundleSize, getCheckpointDurationMillis, getMaxRecordsPerBatch, getMinReadTimeMillis, getReadTimePercentage, getUsesProvidedSparkContext, isCacheDisabled, setBatchIntervalMillis, setBundleSize, setCacheDisabled, setCheckpointDurationMillis, setMaxRecordsPerBatch, setMinReadTimeMillis, setReadTimePercentage, setUsesProvidedSparkContext

Methods inherited from interface org.apache.beam.sdk.options.StreamingOptions

getUpdateCompatibilityVersion, isStreaming, setStreaming, setUpdateCompatibilityVersion

Method Detail

getProvidedSparkContext

org.apache.spark.api.java.JavaSparkContext getProvidedSparkContext()

setProvidedSparkContext

void setProvidedSparkContext(org.apache.spark.api.java.JavaSparkContext jsc)

getListeners

@InstanceFactory(EmptyListenersList.class)
java.util.List<org.apache.spark.streaming.api.java.JavaStreamingListener> getListeners()

setListeners

void setListeners(java.util.List<org.apache.spark.streaming.api.java.JavaStreamingListener> listeners)