org.apache.beam.sdk.io.gcp.spanner.changestreams.action

Class DataChangeRecordAction

java.lang.Object

org.apache.beam.sdk.io.gcp.spanner.changestreams.action.DataChangeRecordAction

public class DataChangeRecordAction
extends java.lang.Object

This class is part of the process for ReadChangeStreamPartitionDoFn SDF. It is responsible for processing DataChangeRecords. The records will simply be emitted to the received output receiver.

Constructor Summary

Constructors

Constructor and Description
DataChangeRecordAction()

Method Summary

Modifier and Type

Method and Description

java.util.Optional<org.apache.beam.sdk.transforms.DoFn.ProcessContinuation>

run(PartitionMetadata partition, DataChangeRecord record, org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<TimestampRange,com.google.cloud.Timestamp> tracker, org.apache.beam.sdk.transforms.DoFn.OutputReceiver<DataChangeRecord> outputReceiver, org.apache.beam.sdk.transforms.splittabledofn.ManualWatermarkEstimator<org.joda.time.Instant> watermarkEstimator, ThroughputEstimator throughputEstimator) This is the main processing function for a DataChangeRecord.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DataChangeRecordAction

public DataChangeRecordAction()

Method Detail

run

public java.util.Optional<org.apache.beam.sdk.transforms.DoFn.ProcessContinuation> run(PartitionMetadata partition,
                                                                                       DataChangeRecord record,
                                                                                       org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<TimestampRange,com.google.cloud.Timestamp> tracker,
                                                                                       org.apache.beam.sdk.transforms.DoFn.OutputReceiver<DataChangeRecord> outputReceiver,
                                                                                       org.apache.beam.sdk.transforms.splittabledofn.ManualWatermarkEstimator<org.joda.time.Instant> watermarkEstimator,
                                                                                       ThroughputEstimator throughputEstimator)

This is the main processing function for a DataChangeRecord. It returns an Optional of DoFn.ProcessContinuation to indicate if the calling function should stop or not. If the Optional returned is empty, it means that the calling function can continue with the processing. If an Optional of DoFn.ProcessContinuation.stop() is returned, it means that this function was unable to claim the timestamp of the DataChangeRecord, so the caller should stop.

When processing the DataChangeRecord the following procedure is applied:

1. We try to cliam the data change record commit timestamp. If it is not possible, we stop here and return.
2. We emit the data change record through the DoFn.OutputReceiver.
3. We update the watermark to the data change record commit timestamp.

Parameters:

partition - the current partition being processed

record - the change stream data record received

tracker - the restriction tracker of the ReadChangeStreamPartitionDoFn SDF

outputReceiver - the output receiver of the ReadChangeStreamPartitionDoFn SDF

watermarkEstimator - the watermark estimator of the ReadChangeStreamPartitionDoFn SDF

throughputEstimator - an estimator to calculate local throughput of the ReadChangeStreamPartitionDoFn.

Returns:

Optional.empty() if the caller can continue processing more records. A non empty Optional with DoFn.ProcessContinuation.stop() if this function was unable to claim the ChildPartitionsRecord timestamp