Package org.apache.beam.examples.complete.kafkatopubsub.transforms

Class FormatTransform

java.lang.Object

org.apache.beam.examples.complete.kafkatopubsub.transforms.FormatTransform

public class FormatTransform
extends java.lang.Object

Different transformations over the processed data in the pipeline.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	FormatTransform.FORMAT
static class	FormatTransform.FormatOutput	The FormatTransform.FormatOutput wraps a String serializable messages with the PubsubMessage class.

Constructor Summary

Constructors

Constructor	Description
FormatTransform()

Method Summary

Modifier and Type	Method	Description
static org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<java.lang.String,AvroDataClass>>>	readAvrosFromKafka(java.lang.String bootstrapServers, java.util.List<java.lang.String> topicsList, java.util.Map<java.lang.String,java.lang.Object> config, java.util.Map<java.lang.String,java.lang.String> sslConfig)	Configures Kafka consumer to read avros to AvroDataClass format.
static org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<java.lang.String,java.lang.String>>>	readFromKafka(java.lang.String bootstrapServers, java.util.List<java.lang.String> topicsList, java.util.Map<java.lang.String,java.lang.Object> kafkaConfig, java.util.Map<java.lang.String,java.lang.String> sslConfig)	Configures Kafka consumer.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FormatTransform

public FormatTransform()

Method Detail

readFromKafka

public static org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<java.lang.String,java.lang.String>>> readFromKafka(java.lang.String bootstrapServers,
                                                                                                                                                                                                                              java.util.List<java.lang.String> topicsList,
                                                                                                                                                                                                                              java.util.Map<java.lang.String,java.lang.Object> kafkaConfig,
                                                                                                                                                                                                                              java.util.Map<java.lang.String,java.lang.String> sslConfig)

Configures Kafka consumer.

Parameters:

bootstrapServers - Kafka servers to read from

topicsList - Kafka topics to read from

kafkaConfig - configuration for the Kafka consumer

sslConfig - configuration for the SSL connection

Returns:

configured reading from Kafka

readAvrosFromKafka

public static org.apache.beam.sdk.transforms.PTransform<org.apache.beam.sdk.values.PBegin,org.apache.beam.sdk.values.PCollection<org.apache.beam.sdk.values.KV<java.lang.String,AvroDataClass>>> readAvrosFromKafka(java.lang.String bootstrapServers,
                                                                                                                                                                                                                                java.util.List<java.lang.String> topicsList,
                                                                                                                                                                                                                                java.util.Map<java.lang.String,java.lang.Object> config,
                                                                                                                                                                                                                                java.util.Map<java.lang.String,java.lang.String> sslConfig)

Configures Kafka consumer to read avros to AvroDataClass format.

Parameters:

bootstrapServers - Kafka servers to read from

topicsList - Kafka topics to read from

config - configuration for the Kafka consumer

Returns:

configured reading from Kafka