Package org.apache.beam.examples.complete.game

Class LeaderBoard

  • Direct Known Subclasses:
    GameStats, StatefulTeamScore
    public class LeaderBoard
extends HourlyTeamScore
    This class is the third in a series of four pipelines that tell a story in a 'gaming' domain, following UserScore and HourlyTeamScore. Concepts include: processing unbounded data using fixed windows; use of custom timestamps and event-time processing; generation of early/speculative results; using .accumulatingFiredPanes() to do cumulative processing of late- arriving data.

    This pipeline processes an unbounded stream of 'game events'. The calculation of the team scores uses fixed windowing based on event time (the time of the game play event), not processing time (the time that an event is processed by the pipeline). The pipeline calculates the sum of scores per team, for each window. By default, the team scores are calculated using one-hour windows.

    In contrast-- to demo another windowing option-- the user scores are calculated using a global window, which periodically (every ten minutes) emits cumulative user score sums.

    In contrast to the previous pipelines in the series, which used static, finite input data, here we're using an unbounded data source, which lets us provide speculative results, and allows handling of late data, at much lower latency. We can use the early/speculative results to keep a 'leaderboard' updated in near-realtime. Our handling of late data lets us generate correct results, e.g. for 'team prizes'. We're now outputting window results as they're calculated, giving us much lower latency than with the previous batch examples.

    Run injector.Injector to generate pubsub data for this pipeline. The Injector documentation provides more detail on how to do this.

    To execute this pipeline, specify the pipeline configuration like this: 

    
 --project=YOUR_PROJECT_ID
 --tempLocation=gs://YOUR_TEMP_DIRECTORY
 --runner=YOUR_RUNNER
 --dataset=YOUR-DATASET
 --topic=projects/YOUR-PROJECT/topics/YOUR-TOPIC

    The BigQuery dataset you specify must already exist. The PubSub topic you specify should be the same topic to which the Injector is publishing.

    • Constructor Detail

      • LeaderBoard

        public LeaderBoard()

    • Method Detail

      • configureWindowedTableWrite

        protected static java.util.Map<java.lang.String,​WriteToBigQuery.FieldInfo<org.apache.beam.sdk.values.KV<java.lang.String,​java.lang.Integer>>> configureWindowedTableWrite()
        Create a map of information that describes how to write pipeline output to BigQuery. This map is used to write team score sums and includes event timing information.

      • configureBigQueryWrite

        protected static java.util.Map<java.lang.String,​WriteToBigQuery.FieldInfo<org.apache.beam.sdk.values.KV<java.lang.String,​java.lang.Integer>>> configureBigQueryWrite()
        Create a map of information that describes how to write pipeline output to BigQuery. This map is passed to the WriteToBigQuery constructor to write user score sums.

      • configureGlobalWindowBigQueryWrite

        protected static java.util.Map<java.lang.String,​WriteToBigQuery.FieldInfo<org.apache.beam.sdk.values.KV<java.lang.String,​java.lang.Integer>>> configureGlobalWindowBigQueryWrite()
        Create a map of information that describes how to write pipeline output to BigQuery. This map is used to write user score sums.

      • main

        public static void main​(java.lang.String[] args)
                 throws java.lang.Exception
        Throws:
        java.lang.Exception