org.apache.beam.examples.cookbook

Class DistinctExample

java.lang.Object

org.apache.beam.examples.cookbook.DistinctExample

public class DistinctExample
extends java.lang.Object

This example uses as input text of King Lear, by William Shakespeare as plaintext files, and will remove any duplicate lines from this file. (The output does not preserve any input order).

Concepts: the Distinct transform, and how to wire transforms together. Demonstrates TextIO.Read/ Distinct/TextIO.Write.

To execute this pipeline locally, specify a local output file or output prefix on GCS: --output=[YOUR_LOCAL_FILE | gs://YOUR_OUTPUT_PREFIX]

To change the runner, specify:

 --runner=YOUR_SELECTED_RUNNER
 

See examples/java/README.md for instructions about how to configure different runners.

The input defaults to gs://apache-beam-samples/shakespeare/kinglear.txt and can be overridden with --input.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	DistinctExample.Options Options supported by DistinctExample.

Constructor Summary

Constructors

Constructor and Description
DistinctExample()

Method Summary

Modifier and Type	Method and Description
static void	main(java.lang.String[] args)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DistinctExample

public DistinctExample()

Method Detail

main

public static void main(java.lang.String[] args)
                 throws java.lang.Exception

Throws:

java.lang.Exception