org.apache.beam.sdk.io.gcp.spanner.changestreams.dofn

Class DetectNewPartitionsDoFn

java.lang.Object

org.apache.beam.sdk.transforms.DoFn<PartitionMetadata,PartitionMetadata>

org.apache.beam.sdk.io.gcp.spanner.changestreams.dofn.DetectNewPartitionsDoFn

All Implemented Interfaces:

java.io.Serializable, org.apache.beam.sdk.transforms.display.HasDisplayData

@DoFn.UnboundedPerElement
public class DetectNewPartitionsDoFn
extends org.apache.beam.sdk.transforms.DoFn<PartitionMetadata,PartitionMetadata>

A SplittableDoFn (SDF) that is responsible for scheduling partitions to be queried. This component will periodically scan the partition metadata table looking for partitions in the PartitionMetadata.State.CREATED, update their state to PartitionMetadata.State.SCHEDULED and output them to the next stage in the pipeline.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.transforms.DoFn

org.apache.beam.sdk.transforms.DoFn.AlwaysFetched, org.apache.beam.sdk.transforms.DoFn.BoundedPerElement, org.apache.beam.sdk.transforms.DoFn.BundleFinalizer, org.apache.beam.sdk.transforms.DoFn.Element, org.apache.beam.sdk.transforms.DoFn.FieldAccess, org.apache.beam.sdk.transforms.DoFn.FinishBundle, org.apache.beam.sdk.transforms.DoFn.FinishBundleContext, org.apache.beam.sdk.transforms.DoFn.GetInitialRestriction, org.apache.beam.sdk.transforms.DoFn.GetInitialWatermarkEstimatorState, org.apache.beam.sdk.transforms.DoFn.GetRestrictionCoder, org.apache.beam.sdk.transforms.DoFn.GetSize, org.apache.beam.sdk.transforms.DoFn.GetWatermarkEstimatorStateCoder, org.apache.beam.sdk.transforms.DoFn.Key, org.apache.beam.sdk.transforms.DoFn.MultiOutputReceiver, org.apache.beam.sdk.transforms.DoFn.NewTracker, org.apache.beam.sdk.transforms.DoFn.NewWatermarkEstimator, org.apache.beam.sdk.transforms.DoFn.OnTimer, org.apache.beam.sdk.transforms.DoFn.OnTimerContext, org.apache.beam.sdk.transforms.DoFn.OnTimerFamily, org.apache.beam.sdk.transforms.DoFn.OnWindowExpiration, org.apache.beam.sdk.transforms.DoFn.OnWindowExpirationContext, org.apache.beam.sdk.transforms.DoFn.OutputReceiver<T>, org.apache.beam.sdk.transforms.DoFn.ProcessContext, org.apache.beam.sdk.transforms.DoFn.ProcessContinuation, org.apache.beam.sdk.transforms.DoFn.ProcessElement, org.apache.beam.sdk.transforms.DoFn.RequiresStableInput, org.apache.beam.sdk.transforms.DoFn.RequiresTimeSortedInput, org.apache.beam.sdk.transforms.DoFn.Restriction, org.apache.beam.sdk.transforms.DoFn.Setup, org.apache.beam.sdk.transforms.DoFn.SideInput, org.apache.beam.sdk.transforms.DoFn.SplitRestriction, org.apache.beam.sdk.transforms.DoFn.StartBundle, org.apache.beam.sdk.transforms.DoFn.StartBundleContext, org.apache.beam.sdk.transforms.DoFn.StateId, org.apache.beam.sdk.transforms.DoFn.Teardown, org.apache.beam.sdk.transforms.DoFn.TimerFamily, org.apache.beam.sdk.transforms.DoFn.TimerId, org.apache.beam.sdk.transforms.DoFn.Timestamp, org.apache.beam.sdk.transforms.DoFn.TruncateRestriction, org.apache.beam.sdk.transforms.DoFn.UnboundedPerElement, org.apache.beam.sdk.transforms.DoFn.WatermarkEstimatorState, org.apache.beam.sdk.transforms.DoFn.WindowedContext

Constructor Summary

Constructors

Constructor and Description
DetectNewPartitionsDoFn(DaoFactory daoFactory, MapperFactory mapperFactory, ActionFactory actionFactory, ChangeStreamMetrics metrics) This class needs a DaoFactory to build DAOs to access the partition metadata tables.

Method Summary

Modifier and Type	Method and Description
org.joda.time.Instant	getInitialWatermarkEstimatorState(PartitionMetadata partition)
double	getSize(TimestampRange restriction)
TimestampRange	initialRestriction(PartitionMetadata partition) Uses an TimestampRange with a max range.
DetectNewPartitionsRangeTracker	newTracker(TimestampRange restriction)
org.apache.beam.sdk.transforms.splittabledofn.ManualWatermarkEstimator<org.joda.time.Instant>	newWatermarkEstimator(org.joda.time.Instant watermarkEstimatorState)
org.apache.beam.sdk.transforms.DoFn.ProcessContinuation	processElement(org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<TimestampRange,com.google.cloud.Timestamp> tracker, org.apache.beam.sdk.transforms.DoFn.OutputReceiver<PartitionMetadata> receiver, org.apache.beam.sdk.transforms.splittabledofn.ManualWatermarkEstimator<org.joda.time.Instant> watermarkEstimator) Main processing function for the DetectNewPartitionsDoFn function.
void	setAveragePartitionBytesSize(long averagePartitionBytesSize) Sets the average partition bytes size to estimate the backlog of this DoFn.
void	setup() Obtains the instance of DetectNewPartitionsAction.

Methods inherited from class org.apache.beam.sdk.transforms.DoFn

getAllowedTimestampSkew, getInputTypeDescriptor, getOutputTypeDescriptor, populateDisplayData, prepareForProcessing

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DetectNewPartitionsDoFn

public DetectNewPartitionsDoFn(DaoFactory daoFactory,
                               MapperFactory mapperFactory,
                               ActionFactory actionFactory,
                               ChangeStreamMetrics metrics)

This class needs a DaoFactory to build DAOs to access the partition metadata tables. It uses mappers to transform database rows into the PartitionMetadata model. It builds the delegating action class using the ActionFactory. It emits metrics for the partitions read using the ChangeStreamMetrics. It re-schedules the process element function to be executed according to the default resume interval as in DEFAULT_RESUME_DURATION (best effort).

Parameters:

daoFactory - the DaoFactory to construct PartitionMetadataDaos

mapperFactory - the MapperFactory to construct PartitionMetadataMappers

actionFactory - the ActionFactory to construct actions

metrics - the ChangeStreamMetrics to emit partition related metrics

Method Detail

getInitialWatermarkEstimatorState

@DoFn.GetInitialWatermarkEstimatorState
public org.joda.time.Instant getInitialWatermarkEstimatorState(@DoFn.Element
                                                                                                       PartitionMetadata partition)

newWatermarkEstimator

@DoFn.NewWatermarkEstimator
public org.apache.beam.sdk.transforms.splittabledofn.ManualWatermarkEstimator<org.joda.time.Instant> newWatermarkEstimator(@DoFn.WatermarkEstimatorState
                                                                                                                                                       org.joda.time.Instant watermarkEstimatorState)

initialRestriction

@DoFn.GetInitialRestriction
public TimestampRange initialRestriction(@DoFn.Element
                                                                     PartitionMetadata partition)

Uses an TimestampRange with a max range. This is because it does not know beforehand how many partitions it will schedule.

Returns:

the timestamp range for the component

getSize

@DoFn.GetSize
public double getSize(@DoFn.Restriction
                                    TimestampRange restriction)

newTracker

@DoFn.NewTracker
public DetectNewPartitionsRangeTracker newTracker(@DoFn.Restriction
                                                                   TimestampRange restriction)

setup

@DoFn.Setup
public void setup()

Obtains the instance of DetectNewPartitionsAction.

processElement

@DoFn.ProcessElement
public org.apache.beam.sdk.transforms.DoFn.ProcessContinuation processElement(org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<TimestampRange,com.google.cloud.Timestamp> tracker,
                                                                                                   org.apache.beam.sdk.transforms.DoFn.OutputReceiver<PartitionMetadata> receiver,
                                                                                                   org.apache.beam.sdk.transforms.splittabledofn.ManualWatermarkEstimator<org.joda.time.Instant> watermarkEstimator)

Main processing function for the DetectNewPartitionsDoFn function. It will delegate to the DetectNewPartitionsAction class.

setAveragePartitionBytesSize

public void setAveragePartitionBytesSize(long averagePartitionBytesSize)

Sets the average partition bytes size to estimate the backlog of this DoFn. Must be called after the initialization of this DoFn.

Parameters:

averagePartitionBytesSize - the estimated average size of a partition record used in the backlog bytes calculation (DoFn.GetSize)