Packages

  • package root
    Definition Classes
    root
  • package com
    Definition Classes
    root
  • package spotify
    Definition Classes
    com
  • package scio
    Definition Classes
    spotify
  • package testing
    Definition Classes
    scio
  • object JobTest

    Set up a Scio job for end-to-end unit testing.

    Set up a Scio job for end-to-end unit testing. To be used in a PipelineSpec. For example:

    import com.spotify.scio.testing._

class WordCountTest extends PipelineSpec {

  // Mock input data, mock distributed cache and expected result
  val inData = Seq("a b c d e", "a b a b")
  val distCache = Map(1 -> "Jan", 2 -> "Feb", 3 -> "Mar")
  val expected = Seq("a: 3", "b: 3", "c: 1", "d: 1", "e: 1")

  // Test specification
  "WordCount" should "work" in {
    JobTest("com.spotify.scio.examples.WordCount")
    // Or the type safe version
    // JobTest[com.spotify.scio.examples.WordCount.type]

      // Command line arguments
      .args("--input=in.txt", "--output=out.txt")

      // Mock input data
      .input(TextIO("in.txt"), inData)

      // Mock distributed cache
      .distCache(DistCacheIO("gs://dataflow-samples/samples/misc/months.txt"), distCache)

      // Verify output
      .output(TextIO("out.txt")) { actual => actual should containInAnyOrder (expected) }

      // Run job test
      .run()
  }
}
    Definition Classes
    testing
  • Builder

class Builder extends AnyRef

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Builder
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new Builder(state: BuilderState)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def args(newArgs: String*): Builder

    Feed command line arguments to the pipeline being tested.

  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def clone(): AnyRef
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  7. def counter(counter: Counter)(assertion: (Long) ⇒ Unit): Builder

    Evaluate a Counter in the pipeline being tested.

    Evaluate a Counter in the pipeline being tested.

    counter

    counter to be evaluated

    assertion

    assertion for the counter result's committed value

  8. def distCache[T](key: DistCacheIO[T], value: T): Builder

    Feed an distributed cache to the pipeline being tested.

    Feed an distributed cache to the pipeline being tested. Note that DistCacheIO[T] must match the one used inside the pipeline, e.g. DistCacheIO[Set[String]]("dc.txt") with sc.distCache("dc.txt")(f => scala.io.Source.fromFile(f).getLines().toSet).

    value

    mock value, must be serializable.

  9. def distCacheFunc[T](key: DistCacheIO[T], initFn: () ⇒ T): Builder

    Feed an distributed cache to the pipeline being tested.

    Feed an distributed cache to the pipeline being tested. Note that DistCacheIO[T] must match the one used inside the pipeline, e.g. DistCacheIO[Set[String]]("dc.txt") with sc.distCache("dc.txt")(f => scala.io.Source.fromFile(f).getLines().toSet).

    initFn

    init function, must be serializable.

  10. def distribution(distribution: Distribution)(assertion: (DistributionResult) ⇒ Unit): Builder

    Evaluate a Distribution in the pipeline being tested.

    Evaluate a Distribution in the pipeline being tested.

    distribution

    distribution to be evaluated

    assertion

    assertion for the distribution result's committed value

  11. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  12. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  13. def finalize(): Unit
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  14. def gauge(gauge: Gauge)(assertion: (GaugeResult) ⇒ Unit): Builder

    Evaluate a Gauge in the pipeline being tested.

    Evaluate a Gauge in the pipeline being tested.

    gauge

    gauge to be evaluated

    assertion

    assertion for the gauge result's committed value

  15. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
  16. def hashCode(): Int
    Definition Classes
    AnyRef → Any
  17. def input[T](key: TestIO[T], value: Iterable[T]): Builder

    Feed an input to the pipeline being tested.

    Feed an input to the pipeline being tested. Note that TestIO[T] must match the one used inside the pipeline, e.g. AvroIO[MyRecord]("in.avro") with sc.avroFile[MyRecord]("in.avro").

  18. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  19. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  20. final def notify(): Unit
    Definition Classes
    AnyRef
  21. final def notifyAll(): Unit
    Definition Classes
    AnyRef
  22. def output[T](key: TestIO[T])(assertion: (SCollection[T]) ⇒ Unit): Builder

    Evaluate an output of the pipeline being tested.

    Evaluate an output of the pipeline being tested. Note that TestIO[T] must match the one used inside the pipeline, e.g. AvroIO[MyRecord]("out.avro") with data.saveAsAvroFile("out.avro") where data is of type SCollection[MyRecord].

    assertion

    assertion for output data. See SCollectionMatchers for available matchers on an SCollection.

  23. def run(): Unit

    Run the pipeline with test wiring.

  24. def setUp(): Unit

    Set up test wiring.

    Set up test wiring. Use this only if you have custom pipeline wiring and are bypassing run. Make sure tearDown is called afterwards.

  25. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  26. def tearDown(): Unit

    Tear down test wiring.

    Tear down test wiring. Use this only if you have custom pipeline wiring and are bypassing run. Make sure setUp is called before.

  27. val testId: String

    Test ID for input and output wiring.

  28. def toString(): String
    Definition Classes
    Builder → AnyRef → Any
  29. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  30. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  31. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from AnyRef

Inherited from Any

Ungrouped