software.amazon.kinesis.retrieval.polling

Class PrefetchRecordsPublisher

java.lang.Object

software.amazon.kinesis.retrieval.polling.PrefetchRecordsPublisher

All Implemented Interfaces:

org.reactivestreams.Publisher<RecordsRetrieved>, RecordsPublisher

public class PrefetchRecordsPublisher
extends Object
implements RecordsPublisher

This is the prefetch caching class, this class spins up a thread if prefetching is enabled. That thread fetches the next set of records and stores it in the cache. The size of the cache is limited by setting maxPendingProcessRecordsInput i.e. the maximum number of GetRecordsResult that the cache can store, maxByteSize i.e. the byte size of the records stored in the cache and maxRecordsCount i.e. the max number of records that should be present in the cache across multiple GetRecordsResult object. If no data is available in the cache, the call from the record processor is blocked till records are retrieved from Kinesis.

There are three threads namely publisher, demand-notifier and ack-notifier which will contend to drain the events to the Subscriber (ShardConsumer in KCL).

Constructor Summary

Constructors

Constructor and Description
PrefetchRecordsPublisher(int maxPendingProcessRecordsInput, int maxByteSize, int maxRecordsCount, int maxRecordsPerCall, GetRecordsRetrievalStrategy getRecordsRetrievalStrategy, ExecutorService executorService, long idleMillisBetweenCalls, MetricsFactory metricsFactory, String operation, String shardId, ThrottlingReporter throttlingReporter) Constructor for the PrefetchRecordsPublisher.
PrefetchRecordsPublisher(int maxPendingProcessRecordsInput, int maxByteSize, int maxRecordsCount, int maxRecordsPerCall, @NonNull GetRecordsRetrievalStrategy getRecordsRetrievalStrategy, @NonNull ExecutorService executorService, long idleMillisBetweenCalls, @NonNull MetricsFactory metricsFactory, @NonNull String operation, @NonNull String shardId, ThrottlingReporter throttlingReporter, long awaitTerminationTimeoutMillis) Constructor for the PrefetchRecordsPublisher.

Method Summary

Modifier and Type	Method and Description
RequestDetails	getLastSuccessfulRequestDetails() Gets last successful request details.
void	notify(RecordsDeliveryAck recordsDeliveryAck) Notify the publisher on receipt of a data event.
void	restartFrom(RecordsRetrieved recordsRetrieved) Restart from the last accepted and processed
void	shutdown() Shutdowns the publisher.
void	start(ExtendedSequenceNumber extendedSequenceNumber, InitialPositionInStreamExtended initialPositionInStreamExtended) Initializes the publisher with where to start processing.
void	subscribe(org.reactivestreams.Subscriber<? super RecordsRetrieved> s)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

PrefetchRecordsPublisher

public PrefetchRecordsPublisher(int maxPendingProcessRecordsInput,
                                int maxByteSize,
                                int maxRecordsCount,
                                int maxRecordsPerCall,
                                @NonNull
                                @NonNull GetRecordsRetrievalStrategy getRecordsRetrievalStrategy,
                                @NonNull
                                @NonNull ExecutorService executorService,
                                long idleMillisBetweenCalls,
                                @NonNull
                                @NonNull MetricsFactory metricsFactory,
                                @NonNull
                                @NonNull String operation,
                                @NonNull
                                @NonNull String shardId,
                                ThrottlingReporter throttlingReporter,
                                long awaitTerminationTimeoutMillis)

Constructor for the PrefetchRecordsPublisher. This cache prefetches records from Kinesis and stores them in a LinkedBlockingQueue.

Parameters:

maxPendingProcessRecordsInput - Max number of ProcessRecordsInput that can be held in the cache before blocking

maxByteSize - Max byte size of the queue before blocking next get records call

maxRecordsCount - Max number of records in the queue across all ProcessRecordInput objects

maxRecordsPerCall - Max records to be returned per call

getRecordsRetrievalStrategy - Retrieval strategy for the get records call

executorService - Executor service for the cache

idleMillisBetweenCalls - maximum time to wait before dispatching the next get records call

awaitTerminationTimeoutMillis - maximum time to wait for graceful shutdown of executorService

See Also:

PrefetchRecordsPublisher

PrefetchRecordsPublisher

public PrefetchRecordsPublisher(int maxPendingProcessRecordsInput,
                                int maxByteSize,
                                int maxRecordsCount,
                                int maxRecordsPerCall,
                                GetRecordsRetrievalStrategy getRecordsRetrievalStrategy,
                                ExecutorService executorService,
                                long idleMillisBetweenCalls,
                                MetricsFactory metricsFactory,
                                String operation,
                                String shardId,
                                ThrottlingReporter throttlingReporter)

Constructor for the PrefetchRecordsPublisher. This cache prefetches records from Kinesis and stores them in a LinkedBlockingQueue.

Parameters:

maxPendingProcessRecordsInput - Max number of ProcessRecordsInput that can be held in the cache before blocking

maxByteSize - Max byte size of the queue before blocking next get records call

maxRecordsCount - Max number of records in the queue across all ProcessRecordInput objects

maxRecordsPerCall - Max records to be returned per call

getRecordsRetrievalStrategy - Retrieval strategy for the get records call

executorService - Executor service for the cache

idleMillisBetweenCalls - maximum time to wait before dispatching the next get records call

See Also:

PrefetchRecordsPublisher

Method Detail

start

public void start(ExtendedSequenceNumber extendedSequenceNumber,
                  InitialPositionInStreamExtended initialPositionInStreamExtended)

Description copied from interface: RecordsPublisher

Initializes the publisher with where to start processing. If there is a stored sequence number the publisher will begin from that sequence number, otherwise it will use the initial position.

Specified by:

start in interface RecordsPublisher

Parameters:

extendedSequenceNumber - the sequence number to start processing from

initialPositionInStreamExtended - if there is no sequence number the initial position to use

shutdown

public void shutdown()

Description copied from interface: RecordsPublisher

Shutdowns the publisher. Once this method returns the publisher should no longer provide any records.

Specified by:

shutdown in interface RecordsPublisher

getLastSuccessfulRequestDetails

public RequestDetails getLastSuccessfulRequestDetails()

Description copied from interface: RecordsPublisher

Gets last successful request details.

Specified by:

getLastSuccessfulRequestDetails in interface RecordsPublisher

Returns:

details associated with last successful request.

restartFrom

public void restartFrom(RecordsRetrieved recordsRetrieved)

Description copied from interface: RecordsPublisher

Restart from the last accepted and processed

Specified by:

restartFrom in interface RecordsPublisher

Parameters:

recordsRetrieved - the processRecordsInput to restart from

subscribe

public void subscribe(org.reactivestreams.Subscriber<? super RecordsRetrieved> s)

Specified by:

subscribe in interface org.reactivestreams.Publisher<RecordsRetrieved>

notify

public void notify(RecordsDeliveryAck recordsDeliveryAck)

Description copied from interface: RecordsPublisher

Notify the publisher on receipt of a data event.

Specified by:

notify in interface RecordsPublisher

Parameters:

recordsDeliveryAck - acknowledgement received from the subscriber.