Package io.pravega.client.stream.impl

Class ReaderGroupImpl

java.lang.Object

io.pravega.client.stream.impl.ReaderGroupImpl

All Implemented Interfaces:

ReaderGroupNotificationListener, ReaderGroup, ReaderGroupMetrics, java.lang.AutoCloseable

public class ReaderGroupImpl
extends java.lang.Object
implements ReaderGroup, ReaderGroupMetrics

Constructor Summary

Constructors

Constructor	Description
ReaderGroupImpl(java.lang.String scope, java.lang.String groupName, SynchronizerConfig synchronizerConfig, Serializer<InitialUpdate<ReaderGroupState>> initSerializer, Serializer<Update<ReaderGroupState>> updateSerializer, SynchronizerClientFactory clientFactory, Controller controller, ConnectionPool connectionPool)

Method Summary

Modifier and Type	Method	Description
protected boolean	canEqual(java.lang.Object other)
void	close()	Closes the reader group, freeing any resources associated with it.
boolean	equals(java.lang.Object o)
java.util.concurrent.CompletableFuture<java.util.Map<Stream,StreamCut>>	generateStreamCuts(java.util.concurrent.ScheduledExecutorService backgroundExecutor)	Generates a StreamCut after co-ordinating with all the readers using StateSynchronizer.
Controller	getController()
Observable<EndOfDataNotification>	getEndOfDataNotifier(java.util.concurrent.ScheduledExecutorService executor)	Get an end of data notifier for a given reader group.
static java.util.Map<Segment,java.lang.Long>	getEndSegmentsForStreams(ReaderGroupConfig config)
java.lang.String	getGroupName()	Returns the name of the group.
SegmentMetadataClientFactory	getMetaFactory()
ReaderGroupMetrics	getMetrics()	Returns metrics for this reader group.
NotifierFactory	getNotifierFactory()
java.util.Set<java.lang.String>	getOnlineReaders()	Returns a set of readerIds for the readers that are considered to be online by the group.
ReaderSegmentDistribution	getReaderSegmentDistribution()	Returns current distribution of number of segments assigned to each reader in the reader group.
java.lang.String	getScope()	Returns the scope of the stream which the group is associated with.
Observable<SegmentNotification>	getSegmentNotifier(java.util.concurrent.ScheduledExecutorService executor)	Get a segment notifier for a given reader group.
static java.util.Map<SegmentWithRange,java.lang.Long>	getSegmentsForStreams(Controller controller, ReaderGroupConfig config)
java.util.Map<Stream,StreamCut>	getStreamCuts()	Returns a StreamCut for each stream that this reader group is reading from.
java.util.Set<java.lang.String>	getStreamNames()	Returns the set of scoped stream names which was used to configure this group.
StateSynchronizer<ReaderGroupState>	getSynchronizer()
int	hashCode()
java.util.concurrent.CompletableFuture<Checkpoint>	initiateCheckpoint(java.lang.String checkpointName, java.util.concurrent.ScheduledExecutorService backgroundExecutor)	Initiate a checkpoint.
void	readerOffline(java.lang.String readerId, Position lastPosition)	Invoked when a reader that was added to the group is no longer consuming events.
void	resetReaderGroup(ReaderGroupConfig config)	Reset a reader group with the provided ReaderGroupConfig.
java.lang.String	toString()
long	unreadBytes()	Returns the number of bytes between the last recorded position of the readers in the ReaderGroup and the end of the stream(s).
void	updateRetentionStreamCut(java.util.Map<Stream,StreamCut> streamCuts)	Update Retention Stream-Cut for Streams in this Reader Group.

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

ReaderGroupImpl

public ReaderGroupImpl(java.lang.String scope,
                       java.lang.String groupName,
                       SynchronizerConfig synchronizerConfig,
                       Serializer<InitialUpdate<ReaderGroupState>> initSerializer,
                       Serializer<Update<ReaderGroupState>> updateSerializer,
                       SynchronizerClientFactory clientFactory,
                       Controller controller,
                       ConnectionPool connectionPool)

Method Detail

updateRetentionStreamCut

public void updateRetentionStreamCut(java.util.Map<Stream,StreamCut> streamCuts)

Description copied from interface: ReaderGroup

Update Retention Stream-Cut for Streams in this Reader Group. See ReaderGroupConfig.StreamDataRetention.MANUAL_RELEASE_AT_USER_STREAMCUT

Specified by:

updateRetentionStreamCut in interface ReaderGroup

Parameters:

streamCuts - A Map with a Stream-Cut for each Stream. StreamCut indicates position in the Stream till which data has been consumed and can be deleted.

readerOffline

public void readerOffline(java.lang.String readerId,
                          Position lastPosition)

Description copied from interface: ReaderGroup

Invoked when a reader that was added to the group is no longer consuming events. This will cause the events that were going to that reader to be redistributed among the other readers. Events after the lastPosition provided will be (re)read by other readers in the ReaderGroup. Note that this method is automatically invoked by EventStreamReader.close()

Specified by:

readerOffline in interface ReaderGroup

Parameters:

readerId - The id of the reader that is offline.

lastPosition - The position of the last event that was successfully processed by the reader.

getOnlineReaders

public java.util.Set<java.lang.String> getOnlineReaders()

Description copied from interface: ReaderGroup

Returns a set of readerIds for the readers that are considered to be online by the group. i.e. EventStreamClientFactory.createReader(String, String, Serializer, ReaderConfig) was called but ReaderGroup.readerOffline(String, Position) was not called subsequently.

Specified by:

getOnlineReaders in interface ReaderGroup

Returns:

Set of active reader IDs of the group

getStreamNames

public java.util.Set<java.lang.String> getStreamNames()

Description copied from interface: ReaderGroup

Returns the set of scoped stream names which was used to configure this group.

Specified by:

getStreamNames in interface ReaderGroup

Returns:

Set of streams for this group.

initiateCheckpoint

public java.util.concurrent.CompletableFuture<Checkpoint> initiateCheckpoint(java.lang.String checkpointName,
                                                                             java.util.concurrent.ScheduledExecutorService backgroundExecutor)

Description copied from interface: ReaderGroup

Initiate a checkpoint. This causes all readers in the group to receive a special EventRead that contains the provided checkpoint name. This can be used to provide an indication to them that they should persist their state. Once all of the readers have received the notification and resumed reading the future will return a Checkpoint object which contains the StreamCut of the reader group at the time they received the checkpoint. This can be used to reset the group to this point in the stream by calling ReaderGroup.resetReaderGroup(ReaderGroupConfig) if the checkpoint fails or the result cannot be obtained an exception will be set on the future. This method can be called and a new checkpoint can be initiated while another is still in progress if they have different names. If this method is called again before the checkpoint has completed with the same name the future returned to the second caller will refer to the same checkpoint object as the first.

Specified by:

initiateCheckpoint in interface ReaderGroup

Parameters:

checkpointName - The name of the checkpoint (For identification purposes)

backgroundExecutor - A threadPool that can be used to poll for the completion of the checkpoint.

Returns:

A future Checkpoint object that can be used to restore the reader group to this position.

resetReaderGroup

public void resetReaderGroup(ReaderGroupConfig config)

Description copied from interface: ReaderGroup

Reset a reader group with the provided ReaderGroupConfig.

- The stream(s) that are part of the reader group can be specified using ReaderGroupConfig.ReaderGroupConfigBuilder.stream(String), ReaderGroupConfig.ReaderGroupConfigBuilder.stream(String, StreamCut) and ReaderGroupConfig.ReaderGroupConfigBuilder.stream(String, StreamCut, StreamCut).

- To reset a reader group to a given checkpoint use ReaderGroupConfig.ReaderGroupConfigBuilder.startFromCheckpoint(Checkpoint) api.

- To reset a reader group to a given StreamCut use ReaderGroupConfig.ReaderGroupConfigBuilder.startFromStreamCuts(Map).

All existing readers will have to call EventStreamClientFactory.createReader(String, String, Serializer, ReaderConfig). If they continue to read events they will eventually encounter an ReinitializationRequiredException .

Specified by:

resetReaderGroup in interface ReaderGroup

Parameters:

config - The new configuration for the ReaderGroup.

getReaderSegmentDistribution

public ReaderSegmentDistribution getReaderSegmentDistribution()

Description copied from interface: ReaderGroup

Returns current distribution of number of segments assigned to each reader in the reader group.

Specified by:

getReaderSegmentDistribution in interface ReaderGroup

Returns:

an instance of ReaderSegmentDistribution which describes the distribution of segments to readers including unassigned segments.

getSegmentsForStreams

public static java.util.Map<SegmentWithRange,java.lang.Long> getSegmentsForStreams(Controller controller,
                                                                                         ReaderGroupConfig config)

getEndSegmentsForStreams

public static java.util.Map<Segment,java.lang.Long> getEndSegmentsForStreams(ReaderGroupConfig config)

getMetrics

public ReaderGroupMetrics getMetrics()

Description copied from interface: ReaderGroup

Returns metrics for this reader group.

Specified by:

getMetrics in interface ReaderGroup

Returns:

a ReaderGroupMetrics object for this reader group.

unreadBytes

public long unreadBytes()

Description copied from interface: ReaderGroupMetrics

Returns the number of bytes between the last recorded position of the readers in the ReaderGroup and the end of the stream(s). Note: This value may be somewhat delayed.

Specified by:

unreadBytes in interface ReaderGroupMetrics

Returns:

The number of unread bytes.

getSegmentNotifier

public Observable<SegmentNotification> getSegmentNotifier(java.util.concurrent.ScheduledExecutorService executor)

Description copied from interface: ReaderGroupNotificationListener

Get a segment notifier for a given reader group.
A segment notifier is triggered when the total number of segments managed by the ReaderGroup changes. During a scale operation segments can be split into multiple or merge into some other segment causing the total number of segments to change. The total number of segments can also change when configuration of the reader group is changed, for example modify the configuration of a reader group to add/remove a stream.

Note:
* In case of a seal stream operation the segments are sealed and the segment has no successors. In this case the notifier is not triggered.

Specified by:

getSegmentNotifier in interface ReaderGroupNotificationListener

Parameters:

executor - executor on which the listeners run.

Returns:

Observable of type SegmentNotification.

getEndOfDataNotifier

public Observable<EndOfDataNotification> getEndOfDataNotifier(java.util.concurrent.ScheduledExecutorService executor)

Description copied from interface: ReaderGroupNotificationListener

Get an end of data notifier for a given reader group.
An end of data notifier is triggered when the readers have read all the data of the stream(s) managed by the reader group. This is useful to process the stream data with a batch job where the application wants to read data of sealed stream(s).

Note:
* In case of a reader group managing streams, where not all streams are sealed, then EndOfDataNotification notifier is never triggered since readers continue reading the unsealed stream once it has completed reading the sealed stream.

Specified by:

getEndOfDataNotifier in interface ReaderGroupNotificationListener

Parameters:

executor - executor on which the listeners run.

Returns:

Observable of type EndOfDataNotification.

getStreamCuts

public java.util.Map<Stream,StreamCut> getStreamCuts()

Description copied from interface: ReaderGroup

Returns a StreamCut for each stream that this reader group is reading from. The stream cut corresponds to the last checkpointed read offsets of the readers, and it can be used by the application as reference to such a position. A more precise StreamCut, with the latest read offsets can be obtained using ReaderGroup.generateStreamCuts(ScheduledExecutorService) API.

Specified by:

getStreamCuts in interface ReaderGroup

Returns:

Map of streams that this group is reading from to the corresponding cuts.

generateStreamCuts

public java.util.concurrent.CompletableFuture<java.util.Map<Stream,StreamCut>> generateStreamCuts(java.util.concurrent.ScheduledExecutorService backgroundExecutor)

Description copied from interface: ReaderGroup

Generates a StreamCut after co-ordinating with all the readers using StateSynchronizer. A StreamCut is generated by using the latest segment read offsets returned by the readers along with unassigned segments (if any). The configuration ReaderGroupConfig.groupRefreshTimeMillis decides the maximum delay by which the readers return the latest read offsets of their assigned segments.

The StreamCut generated by this API can be used by the application as a reference to a position in the stream. This is guaranteed to be greater than or equal to the position of the readers at the point of invocation of the API. The StreamCuts generated can be used to perform bounded processing of the Stream by configuring a ReaderGroup with a ReaderGroupConfig where the StreamCuts are specified as the lower bound and/or upper bounds using the apis ReaderGroupConfig.ReaderGroupConfigBuilder.stream(Stream, StreamCut, StreamCut) or ReaderGroupConfig.ReaderGroupConfigBuilder.stream(Stream, StreamCut) or ReaderGroupConfig.ReaderGroupConfigBuilder.startFromStreamCuts(Map).

Note: Generating a precise StreamCut, for example a StreamCut pointing to end of Q1 across all segments, is difficult as it depends on the configuration ReaderGroupConfig.groupRefreshTimeMillis which decides the duration by which all the readers running on different machines/ processes respond with their latest read offsets. Hence, the StreamCut would point to a position in the Stream which might include events from Q2. The application thus would need to filter out such additional events.

Specified by:

generateStreamCuts in interface ReaderGroup

Parameters:

backgroundExecutor - A thread pool that will be used to poll if the positions from all the readers have been fetched.

Returns:

A future to a Map of Streams (that this group is reading from) to its corresponding cuts.

close

public void close()

Description copied from interface: ReaderGroup

Closes the reader group, freeing any resources associated with it.

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface ReaderGroup

getScope

public java.lang.String getScope()

Description copied from interface: ReaderGroup

Returns the scope of the stream which the group is associated with.

Specified by:

getScope in interface ReaderGroup

Returns:

A scope string

getGroupName

public java.lang.String getGroupName()

Description copied from interface: ReaderGroup

Returns the name of the group.

Specified by:

getGroupName in interface ReaderGroup

Returns:

Reader group name

getController

public Controller getController()

getMetaFactory

public SegmentMetadataClientFactory getMetaFactory()

getSynchronizer

public StateSynchronizer<ReaderGroupState> getSynchronizer()

getNotifierFactory

public NotifierFactory getNotifierFactory()

equals

public boolean equals(java.lang.Object o)

Overrides:

equals in class java.lang.Object

canEqual

protected boolean canEqual(java.lang.Object other)

hashCode

public int hashCode()

Overrides:

hashCode in class java.lang.Object

toString

public java.lang.String toString()

Overrides:

toString in class java.lang.Object