Packages

object SparkSingletons extends Logging

Singletons an initialization code related to Spark.

While Spark provides various getOrCreate methods, this allows WASP to initialize Spark in a cleaner way and in just a few places and access the various entry points in a much more straightforward way.

Linear Supertypes
Logging, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SparkSingletons
  2. Logging
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  10. def getSQLContext: SQLContext

    Returns the SQLContext singleton, or throws an exception if Spark was not initialized.

    Returns the SQLContext singleton, or throws an exception if Spark was not initialized.

    Annotations
    @throws( ... )
    Exceptions thrown

    IllegalStateException if Spark was not already initialized

  11. def getSparkContext: SparkContext

    Returns the SparkContext singleton, or throws an exception if Spark was not initialized.

    Returns the SparkContext singleton, or throws an exception if Spark was not initialized.

    Annotations
    @throws( ... )
    Exceptions thrown

    IllegalStateException if Spark was not already initialized

  12. def getSparkSession: SparkSession

    Returns the SparkSession singleton, or throws an exception if Spark was not initialized.

    Returns the SparkSession singleton, or throws an exception if Spark was not initialized.

    Annotations
    @throws( ... )
    Exceptions thrown

    IllegalStateException if Spark was not already initialized

  13. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  14. def initializeSpark(sparkConfigModel: SparkConfigModel, telemetryConfig: TelemetryConfigModel, kafkaConfigModel: KafkaConfigModel): Boolean

    Try to initialize the SparkSession in the SparkSingleton with the provided configuration.

    Try to initialize the SparkSession in the SparkSingleton with the provided configuration.

    If it does not exist, it will be created using the settings from sparkConfigModel and true will be returned.

    If the SparkSession already exists, nothing will be done, and false will be returned.

    Annotations
    @throws( ... )
    Exceptions thrown

    IllegalStateException if Spark was already initialized but not by using this method

  15. def initializeSpark(sparkConfigModel: SparkConfigModel, telemetryConfig: TelemetryConfigModel, kafkaConfigProxy: KafkaConfigProxy): Boolean

    Wrapper around SparkConfigModel, telemetryConfig: TelemetryConfigModel, kafkaConfigModel: KafkaConfigModel) that uses the default kafka cluster for telemetry

    Wrapper around SparkConfigModel, telemetryConfig: TelemetryConfigModel, kafkaConfigModel: KafkaConfigModel) that uses the default kafka cluster for telemetry

    Annotations
    @throws( ... )
    Exceptions thrown

    IllegalStateException if Spark was already initialized but not by using this method

  16. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  17. val logger: WaspLogger
    Attributes
    protected
    Definition Classes
    Logging
  18. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  19. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  20. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  21. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  22. def toString(): String
    Definition Classes
    AnyRef → Any
  23. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  24. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  25. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped