org.apache.beam.sdk.io.gcp.spanner.changestreams.dofn

Class DetectNewPartitionsDoFn

java.lang.Object

org.apache.beam.sdk.transforms.DoFn<PartitionMetadata,PartitionMetadata>

org.apache.beam.sdk.io.gcp.spanner.changestreams.dofn.DetectNewPartitionsDoFn

All Implemented Interfaces:

java.io.Serializable, org.apache.beam.sdk.transforms.display.HasDisplayData

@DoFn.UnboundedPerElement
public class DetectNewPartitionsDoFn
extends org.apache.beam.sdk.transforms.DoFn<PartitionMetadata,PartitionMetadata>

A SplittableDoFn (SDF) that is responsible for scheduling partitions to be queried. This component will periodically scan the partition metadata table looking for partitions in the PartitionMetadata.State.CREATED, update their state to PartitionMetadata.State.SCHEDULED and output them to the next stage in the pipeline.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.beam.sdk.transforms.DoFn

org.apache.beam.sdk.transforms.DoFn.AlwaysFetched, org.apache.beam.sdk.transforms.DoFn.BoundedPerElement, org.apache.beam.sdk.transforms.DoFn.BundleFinalizer, org.apache.beam.sdk.transforms.DoFn.Element, org.apache.beam.sdk.transforms.DoFn.FieldAccess, org.apache.beam.sdk.transforms.DoFn.FinishBundle, org.apache.beam.sdk.transforms.DoFn.FinishBundleContext, org.apache.beam.sdk.transforms.DoFn.GetInitialRestriction, org.apache.beam.sdk.transforms.DoFn.GetInitialWatermarkEstimatorState, org.apache.beam.sdk.transforms.DoFn.GetRestrictionCoder, org.apache.beam.sdk.transforms.DoFn.GetSize, org.apache.beam.sdk.transforms.DoFn.GetWatermarkEstimatorStateCoder, org.apache.beam.sdk.transforms.DoFn.Key, org.apache.beam.sdk.transforms.DoFn.MultiOutputReceiver, org.apache.beam.sdk.transforms.DoFn.NewTracker, org.apache.beam.sdk.transforms.DoFn.NewWatermarkEstimator, org.apache.beam.sdk.transforms.DoFn.OnTimer, org.apache.beam.sdk.transforms.DoFn.OnTimerContext, org.apache.beam.sdk.transforms.DoFn.OnTimerFamily, org.apache.beam.sdk.transforms.DoFn.OnWindowExpiration, org.apache.beam.sdk.transforms.DoFn.OnWindowExpirationContext, org.apache.beam.sdk.transforms.DoFn.OutputReceiver<T>, org.apache.beam.sdk.transforms.DoFn.ProcessContext, org.apache.beam.sdk.transforms.DoFn.ProcessContinuation, org.apache.beam.sdk.transforms.DoFn.ProcessElement, org.apache.beam.sdk.transforms.DoFn.RequiresStableInput, org.apache.beam.sdk.transforms.DoFn.RequiresTimeSortedInput, org.apache.beam.sdk.transforms.DoFn.Restriction, org.apache.beam.sdk.transforms.DoFn.Setup, org.apache.beam.sdk.transforms.DoFn.SideInput, org.apache.beam.sdk.transforms.DoFn.SplitRestriction, org.apache.beam.sdk.transforms.DoFn.StartBundle, org.apache.beam.sdk.transforms.DoFn.StartBundleContext, org.apache.beam.sdk.transforms.DoFn.StateId, org.apache.beam.sdk.transforms.DoFn.Teardown, org.apache.beam.sdk.transforms.DoFn.TimerFamily, org.apache.beam.sdk.transforms.DoFn.TimerId, org.apache.beam.sdk.transforms.DoFn.Timestamp, org.apache.beam.sdk.transforms.DoFn.TruncateRestriction, org.apache.beam.sdk.transforms.DoFn.UnboundedPerElement, org.apache.beam.sdk.transforms.DoFn.WatermarkEstimatorState, org.apache.beam.sdk.transforms.DoFn.WindowedContext

Constructor Summary

Constructors

Constructor and Description
DetectNewPartitionsDoFn(DaoFactory daoFactory, MapperFactory mapperFactory, ActionFactory actionFactory, ChangeStreamMetrics metrics) This class needs a DaoFactory to build DAOs to access the partition metadata tables.

Method Summary

Modifier and Type	Method and Description
org.joda.time.Instant	getInitialWatermarkEstimatorState(PartitionMetadata partition)
org.apache.beam.sdk.io.range.OffsetRange	initialRestriction(PartitionMetadata partition) Uses an OffsetRange with a max range.
org.apache.beam.sdk.transforms.splittabledofn.ManualWatermarkEstimator<org.joda.time.Instant>	newWatermarkEstimator(org.joda.time.Instant watermarkEstimatorState)
org.apache.beam.sdk.transforms.DoFn.ProcessContinuation	processElement(org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<org.apache.beam.sdk.io.range.OffsetRange,java.lang.Long> tracker, org.apache.beam.sdk.transforms.DoFn.OutputReceiver<PartitionMetadata> receiver, org.apache.beam.sdk.transforms.splittabledofn.ManualWatermarkEstimator<org.joda.time.Instant> watermarkEstimator) Main processing function for the DetectNewPartitionsDoFn function.
org.apache.beam.sdk.transforms.splittabledofn.OffsetRangeTracker	restrictionTracker(org.apache.beam.sdk.io.range.OffsetRange restriction)
void	setup() Obtains the instance of DetectNewPartitionsAction.

Methods inherited from class org.apache.beam.sdk.transforms.DoFn

getAllowedTimestampSkew, getInputTypeDescriptor, getOutputTypeDescriptor, populateDisplayData, prepareForProcessing

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DetectNewPartitionsDoFn

public DetectNewPartitionsDoFn(DaoFactory daoFactory,
                               MapperFactory mapperFactory,
                               ActionFactory actionFactory,
                               ChangeStreamMetrics metrics)

This class needs a DaoFactory to build DAOs to access the partition metadata tables. It uses mappers to transform database rows into the PartitionMetadata model. It builds the delegating action class using the ActionFactory. It emits metrics for the partitions read using the ChangeStreamMetrics. It re-schedules the process element function to be executed according to the default resume interval as in DEFAULT_RESUME_DURATION (best effort).

Parameters:

daoFactory - the DaoFactory to construct PartitionMetadataDaos

mapperFactory - the MapperFactory to construct PartitionMetadataMappers

actionFactory - the ActionFactory to construct actions

metrics - the ChangeStreamMetrics to emit partition related metrics

Method Detail

getInitialWatermarkEstimatorState

@DoFn.GetInitialWatermarkEstimatorState
public org.joda.time.Instant getInitialWatermarkEstimatorState(@DoFn.Element
                                                                                                       PartitionMetadata partition)

newWatermarkEstimator

@DoFn.NewWatermarkEstimator
public org.apache.beam.sdk.transforms.splittabledofn.ManualWatermarkEstimator<org.joda.time.Instant> newWatermarkEstimator(@DoFn.WatermarkEstimatorState
                                                                                                                                                       org.joda.time.Instant watermarkEstimatorState)

initialRestriction

@DoFn.GetInitialRestriction
public org.apache.beam.sdk.io.range.OffsetRange initialRestriction(@DoFn.Element
                                                                                               PartitionMetadata partition)

Uses an OffsetRange with a max range. This is because it does not know beforehand how many partitions it will schedule.

Returns:

the offset range for the component

restrictionTracker

@DoFn.NewTracker
public org.apache.beam.sdk.transforms.splittabledofn.OffsetRangeTracker restrictionTracker(@DoFn.Restriction
                                                                                                            org.apache.beam.sdk.io.range.OffsetRange restriction)

setup

@DoFn.Setup
public void setup()

Obtains the instance of DetectNewPartitionsAction.

processElement

@DoFn.ProcessElement
public org.apache.beam.sdk.transforms.DoFn.ProcessContinuation processElement(org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker<org.apache.beam.sdk.io.range.OffsetRange,java.lang.Long> tracker,
                                                                                                   org.apache.beam.sdk.transforms.DoFn.OutputReceiver<PartitionMetadata> receiver,
                                                                                                   org.apache.beam.sdk.transforms.splittabledofn.ManualWatermarkEstimator<org.joda.time.Instant> watermarkEstimator)

Main processing function for the DetectNewPartitionsDoFn function. It will delegate to the DetectNewPartitionsAction class.