Value Members

1.  final def !=(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

2.  final def ##(): Int

   Definition Classes

   AnyRef → Any

3.  final def ==(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

4.  final def asInstanceOf[T0]: T0

   Definition Classes

   Any

5.  def clone(): AnyRef

   Attributes

   protected[lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( ... ) @native()

6.  final def eq(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

7.  def equals(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

8.  def finalize(): Unit

   Attributes

   protected[lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( classOf[java.lang.Throwable] )

9.  final def getClass(): Class[_]

   Definition Classes

   AnyRef → Any

   Annotations

   @native()

10.  def hashCode(): Int

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

11.  final def isInstanceOf[T0]: Boolean

    Definition Classes

    Any

12.  def main(args: Array[String]): Unit
13.  final def ne(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

14.  final def notify(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

15.  final def notifyAll(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

16.  def run(pipeline: ETLPipeline)(implicit spark: SparkSession, logger: Logger, arcContext: ARCContext): Option[DataFrame]

    An ETL Pipeline submits each of its stages in order to Spark.

    An ETL Pipeline submits each of its stages in order to Spark. The submission order will match the order declared in the pipeline configuration. Spark may alter the order of evaulation once it has analyzed the DAG. The run method is designed to mimic a basic interpreter, we new stage types are created they need to be added here in order for them to be executed.

    It would be possible to extend this process to support other compute engines as the submitted stages are not specific to Spark.

17.  final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes

    AnyRef

18.  def toString(): String

    Definition Classes

    AnyRef → Any

19.  final def wait(): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

20.  final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

21.  final def wait(arg0: Long): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... ) @native()