Streaming
Related Doc:
package sparkka
Returns an InputDStream of type FlowElementType along with a Flow map element that you can use to attach to your flow.
Returns an InputDStream of type FlowElementType along with a Flow map element that you can use to attach to your flow.
NOTE ensure that you have remoting set up properly in your application.conf; see http://doc.akka.io/docs/akka/snapshot/scala/remoting.html
Backpressures if and when the initial buffer is filled and your input stream has not yet started; in this event, the buffer is held constant up until the provided initial buffer wait duration, after which point we start dropping oldest buffer entries, and backpressure stops.
Name of the receiver actor (default: "akka-stream-receiver-$uuid")
In the event that the InputStream is not yet started, how many elements from the Akka stream should be buffered before dropping oldest entries (default: 50000)
In the event that the InputStream is not yet started and the buffer is full, how long to "block" for until we begin culling oldest items from the buffer (default: 15 seconds)
RDD storage level (default: StorageLevel.MEMORY_AND_DISK_SER_2) for the receiver
Supervisor strategy for the receiver actor (default: ActorSupervisorStrategy.defaultStrategy)
// format: OFF
implicit val actorSystem = ActorSystem() implicit val materializer = ActorMaterializer() implicit val ssc = LocalContext.ssc // InputDStream can then be used to build elements of the graph that require integration with Spark val (inputDStream, feedDInput) = Streaming.streamConnection[Int]() val source = Source.fromGraph(GraphDSL.create() { implicit builder => import GraphDSL.Implicits._ val source = Source(1 to 10) val bCast = builder.add(Broadcast[Int](2)) val merge = builder.add(Merge[Int](2)) val add1 = Flow[Int].map(_ + 1) val times3 = Flow[Int].map(_ * 3) source ~> bCast ~> add1 ~> merge bCast ~> times3 ~> feedDInput ~> merge SourceShape(merge.out) })
// format: ON
Created by Lloyd on 2/28/16.