Class SparkContextFactory
- java.lang.Object
-
- org.apache.beam.runners.spark.translation.SparkContextFactory
-
public final class SparkContextFactory extends java.lang.Object
-
-
Field Summary
Fields Modifier and Type Field Description static java.lang.StringTEST_REUSE_SPARK_CONTEXTDeprecated.This will leak your SparkContext, any attempt to create a new SparkContext later will fail.
-
Method Summary
All Methods Static Methods Concrete Methods Modifier and Type Method Description static voidclearProvidedSparkContext()static org.apache.spark.api.java.JavaSparkContextgetSparkContext(SparkPipelineOptions options)static voidsetProvidedSparkContext(org.apache.spark.api.java.JavaSparkContext providedSparkContext)Set an externally managedJavaSparkContextthat will be used ifSparkPipelineOptions.getUsesProvidedSparkContext()is set totrue.static voidstopSparkContext(org.apache.spark.api.java.JavaSparkContext context)
-
-
-
Field Detail
-
TEST_REUSE_SPARK_CONTEXT
@Deprecated public static final java.lang.String TEST_REUSE_SPARK_CONTEXT
Deprecated.This will leak your SparkContext, any attempt to create a new SparkContext later will fail. Please usesetProvidedSparkContext(JavaSparkContext)/clearProvidedSparkContext()instead to properly control the lifecycle of your context. Alternatively you may also provide a SparkContext usingSparkPipelineOptions.setUsesProvidedSparkContext(boolean)together withSparkContextOptions.setProvidedSparkContext(JavaSparkContext)and close that one appropriately. Tests of this module should useSparkContextRule.If the propertybeam.spark.test.reuseSparkContextis set totruethen the Spark context will be reused for beam pipelines. This property should only be enabled for tests.- See Also:
- Constant Field Values
-
-
Method Detail
-
setProvidedSparkContext
public static void setProvidedSparkContext(org.apache.spark.api.java.JavaSparkContext providedSparkContext)
Set an externally managedJavaSparkContextthat will be used ifSparkPipelineOptions.getUsesProvidedSparkContext()is set totrue.A Spark context can also be provided using
SparkContextOptions.setProvidedSparkContext(JavaSparkContext). However, it will be dropped during serialization potentially leading to confusing behavior. This is particularly the case when used in tests withTestPipeline.
-
clearProvidedSparkContext
public static void clearProvidedSparkContext()
-
getSparkContext
public static org.apache.spark.api.java.JavaSparkContext getSparkContext(SparkPipelineOptions options)
-
stopSparkContext
public static void stopSparkContext(org.apache.spark.api.java.JavaSparkContext context)
-
-