Package org.apache.beam.runners.spark.translation

Class SparkContextFactory

java.lang.Object

org.apache.beam.runners.spark.translation.SparkContextFactory

public final class SparkContextFactory
extends java.lang.Object

Field Summary

Fields

Modifier and Type	Field	Description
static java.lang.String	TEST_REUSE_SPARK_CONTEXT	Deprecated. This will leak your SparkContext, any attempt to create a new SparkContext later will fail.

Method Summary

Modifier and Type	Method	Description
static void	clearProvidedSparkContext()
static org.apache.spark.api.java.JavaSparkContext	getSparkContext(SparkPipelineOptions options)
static void	setProvidedSparkContext(org.apache.spark.api.java.JavaSparkContext providedSparkContext)	Set an externally managed JavaSparkContext that will be used if SparkPipelineOptions.getUsesProvidedSparkContext() is set to true.
static void	stopSparkContext(org.apache.spark.api.java.JavaSparkContext context)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TEST_REUSE_SPARK_CONTEXT

@Deprecated
public static final java.lang.String TEST_REUSE_SPARK_CONTEXT

Deprecated.

This will leak your SparkContext, any attempt to create a new SparkContext later will fail. Please use setProvidedSparkContext(JavaSparkContext) / clearProvidedSparkContext() instead to properly control the lifecycle of your context. Alternatively you may also provide a SparkContext using SparkPipelineOptions.setUsesProvidedSparkContext(boolean) together with SparkContextOptions.setProvidedSparkContext(JavaSparkContext) and close that one appropriately. Tests of this module should use SparkContextRule.

If the property beam.spark.test.reuseSparkContext is set to true then the Spark context will be reused for beam pipelines. This property should only be enabled for tests.

See Also:

Constant Field Values

Method Detail

setProvidedSparkContext

public static void setProvidedSparkContext(org.apache.spark.api.java.JavaSparkContext providedSparkContext)

Set an externally managed JavaSparkContext that will be used if SparkPipelineOptions.getUsesProvidedSparkContext() is set to true.

A Spark context can also be provided using SparkContextOptions.setProvidedSparkContext(JavaSparkContext). However, it will be dropped during serialization potentially leading to confusing behavior. This is particularly the case when used in tests with TestPipeline.

clearProvidedSparkContext

public static void clearProvidedSparkContext()

getSparkContext

public static org.apache.spark.api.java.JavaSparkContext getSparkContext(SparkPipelineOptions options)

stopSparkContext

public static void stopSparkContext(org.apache.spark.api.java.JavaSparkContext context)