org.apache.beam.sdk.io.gcp.pubsub

Class PubsubUnboundedSink

java.lang.Object

org.apache.beam.sdk.transforms.PTransform<PCollection<PubsubMessage>,PDone>

org.apache.beam.sdk.io.gcp.pubsub.PubsubUnboundedSink

All Implemented Interfaces:

Serializable, HasDisplayData

public class PubsubUnboundedSink
extends PTransform<PCollection<PubsubMessage>,PDone>

A PTransform which streams messages to Pubsub.

• The underlying implementation is just a GroupByKey followed by a ParDo which publishes as a side effect. (In the future we want to design and switch to a custom UnboundedSink implementation so as to gain access to system watermark and end-of-pipeline cleanup.)
• We try to send messages in batches while also limiting send latency.
• No stats are logged. Rather some counters are used to keep track of elements and batches.
• Though some background threads are used by the underlying netty system all actual Pubsub calls are blocking. We rely on the underlying runner to allow multiple DoFn instances to execute concurrently and hide latency.
• A failed bundle will cause messages to be resent. Thus we rely on the Pubsub consumer to dedup messages.

See Also:

Serialized Form

Field Summary

Fields inherited from class org.apache.beam.sdk.transforms.PTransform

name

Constructor Summary

Constructors

Constructor and Description
PubsubUnboundedSink(PubsubClient.PubsubClientFactory pubsubFactory, ValueProvider<PubsubClient.TopicPath> topic, String timestampAttribute, String idAttribute, int numShards)

Method Summary

Modifier and Type	Method and Description
PDone	expand(PCollection<PubsubMessage> input)
String	getIdAttribute() Get the id attribute.
String	getTimestampAttribute() Get the timestamp attribute.
PubsubClient.TopicPath	getTopic() Get the topic being written to.
ValueProvider<PubsubClient.TopicPath>	getTopicProvider() Get the ValueProvider for the topic being written to.

Methods inherited from class org.apache.beam.sdk.transforms.PTransform

getAdditionalInputs, getDefaultOutputCoder, getDefaultOutputCoder, getDefaultOutputCoder, getKindString, getName, populateDisplayData, toString, validate

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

PubsubUnboundedSink

public PubsubUnboundedSink(PubsubClient.PubsubClientFactory pubsubFactory,
                           ValueProvider<PubsubClient.TopicPath> topic,
                           String timestampAttribute,
                           String idAttribute,
                           int numShards)

Method Detail

getTopic

public PubsubClient.TopicPath getTopic()

Get the topic being written to.

getTopicProvider

public ValueProvider<PubsubClient.TopicPath> getTopicProvider()

Get the ValueProvider for the topic being written to.

getTimestampAttribute

@Nullable
public String getTimestampAttribute()

Get the timestamp attribute.

getIdAttribute

@Nullable
public String getIdAttribute()

Get the id attribute.

expand

public PDone expand(PCollection<PubsubMessage> input)

Specified by:

expand in class PTransform<PCollection<PubsubMessage>,PDone>