Packages

c

ai.chronon.flink

FlinkJob

class FlinkJob[T] extends AnyRef

Flink job that processes a single streaming GroupBy and writes out the results to the KV store.

There are two versions of the job, tiled and untiled. The untiled version writes out raw events while the tiled version writes out pre-aggregates. See the runGroupByJob and runTiledGroupByJob methods for more details.

T

- The input data type

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. FlinkJob
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new FlinkJob(eventSrc: FlinkSource[T], sinkFn: RichAsyncFunction[PutRequest, WriteResponse], groupByServingInfoParsed: GroupByServingInfoParsed, encoder: Encoder[T], parallelism: Int)

    eventSrc

    - Provider of a Flink Datastream[T] for the given topic and feature group

    sinkFn

    - Async Flink writer function to help us write to the KV store

    groupByServingInfoParsed

    - The GroupBy we are working with

    encoder

    - Spark Encoder for the input data type

    parallelism

    - Parallelism to use for the Flink job

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  8. val exprEval: SparkExpressionEvalFn[T]
    Attributes
    protected
  9. val featureGroupName: String
  10. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  11. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  13. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  14. val kafkaTopic: String
  15. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  16. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  17. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  18. def runGroupByJob(env: StreamExecutionEnvironment): DataStream[WriteResponse]

    The "untiled" version of the Flink app.

    The "untiled" version of the Flink app.

    At a high level, the operators are structured as follows: Kafka source -> Spark expression eval -> Avro conversion -> KV store writer Kafka source - Reads objects of type T (specific case class, Thrift / Proto) from a Kafka topic Spark expression eval - Evaluates the Spark SQL expression in the GroupBy and projects and filters the input data Avro conversion - Converts the Spark expr eval output to a form that can be written out to the KV store (PutRequest object) KV store writer - Writes the PutRequest objects to the KV store using the AsyncDataStream API

    In this untiled version, there are no shuffles and thus this ends up being a single node in the Flink DAG (with the above 4 operators and parallelism as injected by the user).

  19. def runTiledGroupByJob(env: StreamExecutionEnvironment): DataStream[WriteResponse]

    The "tiled" version of the Flink app.

    The "tiled" version of the Flink app.

    The operators are structured as follows:

    1. Kafka source - Reads objects of type T (specific case class, Thrift / Proto) from a Kafka topic 2. Spark expression eval - Evaluates the Spark SQL expression in the GroupBy and projects and filters the input data 3. Window/tiling - This window aggregates incoming events, keeps track of the IRs, and sends them forward so they are written out to the KV store 4. Avro conversion - Finishes converting the output of the window (the IRs) to a form that can be written out to the KV store (PutRequest object) 5. KV store writer - Writes the PutRequest objects to the KV store using the AsyncDataStream API

    The window causes a split in the Flink DAG, so there are two nodes, (1+2) and (3+4+5).

  20. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  21. def toString(): String
    Definition Classes
    AnyRef → Any
  22. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  23. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  24. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from AnyRef

Inherited from Any

Ungrouped