com.amazonaws.services.kinesis.clientlibrary.lib.worker

Class RecordProcessorCheckpointer

java.lang.Object

com.amazonaws.services.kinesis.clientlibrary.lib.worker.RecordProcessorCheckpointer

All Implemented Interfaces:

IRecordProcessorCheckpointer

public class RecordProcessorCheckpointer
extends Object
implements IRecordProcessorCheckpointer

This class is used to enable RecordProcessors to checkpoint their progress. The Amazon Kinesis Client Library will instantiate an object and provide a reference to the application RecordProcessor instance. Amazon Kinesis Client Library will create one instance per shard assignment.

Constructor Summary

Constructors

Constructor and Description
RecordProcessorCheckpointer(ShardInfo shardInfo, ICheckpoint checkpoint, SequenceNumberValidator validator, IMetricsFactory metricsFactory) Only has package level access, since only the Amazon Kinesis Client Library should be creating these.

Method Summary

Modifier and Type	Method and Description
void	checkpoint() This method will checkpoint the progress at the last data record that was delivered to the record processor.
void	checkpoint(com.amazonaws.services.kinesis.model.Record record) This method will checkpoint the progress at the provided record.
void	checkpoint(String sequenceNumber) This method will checkpoint the progress at the provided sequenceNumber.
void	checkpoint(String sequenceNumber, long subSequenceNumber) This method will checkpoint the progress at the provided sequenceNumber and subSequenceNumber, the latter for aggregated records produced with the Producer Library.
ExtendedSequenceNumber	getLargestPermittedCheckpointValue() Used for testing.
ExtendedSequenceNumber	getLastCheckpointValue()
IPreparedCheckpointer	prepareCheckpoint() This method will record a pending checkpoint at the last data record that was delivered to the record processor.
IPreparedCheckpointer	prepareCheckpoint(com.amazonaws.services.kinesis.model.Record record) This method will record a pending checkpoint at the at the provided record.
IPreparedCheckpointer	prepareCheckpoint(String sequenceNumber) This method will record a pending checkpoint at the provided sequenceNumber.
IPreparedCheckpointer	prepareCheckpoint(String sequenceNumber, long subSequenceNumber) This method will record a pending checkpoint at the provided sequenceNumber and subSequenceNumber, the latter for aggregated records produced with the Producer Library.
void	setLargestPermittedCheckpointValue(ExtendedSequenceNumber largestPermittedCheckpointValue)
void	setSequenceNumberAtShardEnd(ExtendedSequenceNumber extendedSequenceNumber) Used to remember the last extended sequence number before SHARD_END to allow us to prevent the checkpointer from checkpointing at the end of the shard twice (i.e.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RecordProcessorCheckpointer

public RecordProcessorCheckpointer(ShardInfo shardInfo,
                                   ICheckpoint checkpoint,
                                   SequenceNumberValidator validator,
                                   IMetricsFactory metricsFactory)

Only has package level access, since only the Amazon Kinesis Client Library should be creating these.

Parameters:

checkpoint - Used to checkpoint progress of a RecordProcessor

validator - Used for validating sequence numbers

Method Detail

checkpoint

public void checkpoint()
                throws KinesisClientLibDependencyException,
                       InvalidStateException,
                       ThrottlingException,
                       ShutdownException

This method will checkpoint the progress at the last data record that was delivered to the record processor. Upon fail over (after a successful checkpoint() call), the new/replacement RecordProcessor instance will receive data records whose sequenceNumber > checkpoint position (for each partition key). In steady state, applications should checkpoint periodically (e.g. once every 5 minutes). Calling this API too frequently can slow down the application (because it puts pressure on the underlying checkpoint storage layer).

Specified by:

checkpoint in interface IRecordProcessorCheckpointer

Throws:

KinesisClientLibDependencyException - Encountered an issue when storing the checkpoint. The application can backoff and retry.

InvalidStateException - Can't store checkpoint. Unable to store the checkpoint in the DynamoDB table (e.g. table doesn't exist).

ThrottlingException - Can't store checkpoint. Can be caused by checkpointing too frequently. Consider increasing the throughput/capacity of the checkpoint store or reducing checkpoint frequency.

ShutdownException - The record processor instance has been shutdown. Another instance may have started processing some of these records already. The application should abort processing via this RecordProcessor instance.

checkpoint

public void checkpoint(com.amazonaws.services.kinesis.model.Record record)
                throws KinesisClientLibDependencyException,
                       InvalidStateException,
                       ThrottlingException,
                       ShutdownException,
                       IllegalArgumentException

This method will checkpoint the progress at the provided record. This method is analogous to IRecordProcessorCheckpointer.checkpoint() but provides the ability to specify the record at which to checkpoint.

Specified by:

checkpoint in interface IRecordProcessorCheckpointer

Parameters:

record - A record at which to checkpoint in this shard. Upon failover, the Kinesis Client Library will start fetching records after this record's sequence number.

Throws:

KinesisClientLibDependencyException - Encountered an issue when storing the checkpoint. The application can backoff and retry.

InvalidStateException - Can't store checkpoint. Unable to store the checkpoint in the DynamoDB table (e.g. table doesn't exist).

ThrottlingException - Can't store checkpoint. Can be caused by checkpointing too frequently. Consider increasing the throughput/capacity of the checkpoint store or reducing checkpoint frequency.

ShutdownException - The record processor instance has been shutdown. Another instance may have started processing some of these records already. The application should abort processing via this RecordProcessor instance.

IllegalArgumentException

checkpoint

public void checkpoint(String sequenceNumber)
                throws KinesisClientLibDependencyException,
                       InvalidStateException,
                       ThrottlingException,
                       ShutdownException,
                       IllegalArgumentException

This method will checkpoint the progress at the provided sequenceNumber. This method is analogous to IRecordProcessorCheckpointer.checkpoint() but provides the ability to specify the sequence number at which to checkpoint.

Specified by:

checkpoint in interface IRecordProcessorCheckpointer

Parameters:

sequenceNumber - A sequence number at which to checkpoint in this shard. Upon failover, the Kinesis Client Library will start fetching records after this sequence number.

Throws:

KinesisClientLibDependencyException - Encountered an issue when storing the checkpoint. The application can backoff and retry.

InvalidStateException - Can't store checkpoint. Unable to store the checkpoint in the DynamoDB table (e.g. table doesn't exist).

ThrottlingException - Can't store checkpoint. Can be caused by checkpointing too frequently. Consider increasing the throughput/capacity of the checkpoint store or reducing checkpoint frequency.

ShutdownException - The record processor instance has been shutdown. Another instance may have started processing some of these records already. The application should abort processing via this RecordProcessor instance.

IllegalArgumentException - The sequence number is invalid for one of the following reasons: 1.) It appears to be out of range, i.e. it is smaller than the last check point value, or larger than the greatest sequence number seen by the associated record processor. 2.) It is not a valid sequence number for a record in this shard.

checkpoint

public void checkpoint(String sequenceNumber,
                       long subSequenceNumber)
                throws KinesisClientLibDependencyException,
                       InvalidStateException,
                       ThrottlingException,
                       ShutdownException,
                       IllegalArgumentException

This method will checkpoint the progress at the provided sequenceNumber and subSequenceNumber, the latter for aggregated records produced with the Producer Library. This method is analogous to IRecordProcessorCheckpointer.checkpoint() but provides the ability to specify the sequence and subsequence numbers at which to checkpoint.

Specified by:

checkpoint in interface IRecordProcessorCheckpointer

Parameters:

sequenceNumber - A sequence number at which to checkpoint in this shard. Upon failover, the Kinesis Client Library will start fetching records after the given sequence and subsequence numbers.

subSequenceNumber - A subsequence number at which to checkpoint within this shard. Upon failover, the Kinesis Client Library will start fetching records after the given sequence and subsequence numbers.

Throws:

KinesisClientLibDependencyException - Encountered an issue when storing the checkpoint. The application can backoff and retry.

InvalidStateException - Can't store checkpoint. Unable to store the checkpoint in the DynamoDB table (e.g. table doesn't exist).

ThrottlingException - Can't store checkpoint. Can be caused by checkpointing too frequently. Consider increasing the throughput/capacity of the checkpoint store or reducing checkpoint frequency.

ShutdownException - The record processor instance has been shutdown. Another instance may have started processing some of these records already. The application should abort processing via this RecordProcessor instance.

IllegalArgumentException - The sequence number is invalid for one of the following reasons: 1.) It appears to be out of range, i.e. it is smaller than the last check point value, or larger than the greatest sequence number seen by the associated record processor. 2.) It is not a valid sequence number for a record in this shard.

prepareCheckpoint

public IPreparedCheckpointer prepareCheckpoint()
                                        throws KinesisClientLibDependencyException,
                                               InvalidStateException,
                                               ThrottlingException,
                                               ShutdownException

This method will record a pending checkpoint at the last data record that was delivered to the record processor. If the application fails over between calling prepareCheckpoint() and checkpoint(), the init() method of the next IRecordProcessor for this shard will be informed of the prepared sequence number Application should use this to assist with idempotency across failover by calling prepareCheckpoint before having side effects, then by calling checkpoint on the returned PreparedCheckpointer after side effects are complete. Use the sequence number passed in to init() to behave idempotently.

Specified by:

prepareCheckpoint in interface IRecordProcessorCheckpointer

Returns:

an IPreparedCheckpointer object that can be called later to persist the checkpoint.

Throws:

KinesisClientLibDependencyException - Encountered an issue when storing the pending checkpoint. The application can backoff and retry.

InvalidStateException - Can't store pending checkpoint. Unable to store the checkpoint in the DynamoDB table (e.g. table doesn't exist).

ThrottlingException - Can't store pending checkpoint. Can be caused by checkpointing too frequently. Consider increasing the throughput/capacity of the checkpoint store or reducing checkpoint frequency.

ShutdownException - The record processor instance has been shutdown. Another instance may have started processing some of these records already. The application should abort processing via this RecordProcessor instance.

prepareCheckpoint

public IPreparedCheckpointer prepareCheckpoint(com.amazonaws.services.kinesis.model.Record record)
                                        throws KinesisClientLibDependencyException,
                                               InvalidStateException,
                                               ThrottlingException,
                                               ShutdownException

This method will record a pending checkpoint at the at the provided record. This method is analogous to IRecordProcessorCheckpointer.prepareCheckpoint() but provides the ability to specify the record at which to prepare the checkpoint.

Specified by:

prepareCheckpoint in interface IRecordProcessorCheckpointer

Parameters:

record - A record at which to prepare checkpoint in this shard. Application should use this to assist with idempotency across failover by calling prepareCheckpoint before having side effects, then by calling checkpoint on the returned PreparedCheckpointer after side effects are complete. Use the sequence number and application state passed in to init() to behave idempotently.

Returns:

an IPreparedCheckpointer object that can be called later to persist the checkpoint.

Throws:

KinesisClientLibDependencyException - Encountered an issue when storing the pending checkpoint. The application can backoff and retry.

InvalidStateException - Can't store pending checkpoint. Unable to store the checkpoint in the DynamoDB table (e.g. table doesn't exist).

ThrottlingException - Can't store pending checkpoint. Can be caused by checkpointing too frequently. Consider increasing the throughput/capacity of the checkpoint store or reducing checkpoint frequency.

ShutdownException - The record processor instance has been shutdown. Another instance may have started processing some of these records already. The application should abort processing via this RecordProcessor instance.

prepareCheckpoint

public IPreparedCheckpointer prepareCheckpoint(String sequenceNumber)
                                        throws KinesisClientLibDependencyException,
                                               InvalidStateException,
                                               ThrottlingException,
                                               ShutdownException

This method will record a pending checkpoint at the provided sequenceNumber. This method is analogous to IRecordProcessorCheckpointer.prepareCheckpoint() but provides the ability to specify the sequence number at which to checkpoint.

Specified by:

prepareCheckpoint in interface IRecordProcessorCheckpointer

Parameters:

sequenceNumber - A sequence number at which to prepare checkpoint in this shard.

Returns:

an IPreparedCheckpointer object that can be called later to persist the checkpoint.

Throws:

KinesisClientLibDependencyException - Encountered an issue when storing the pending checkpoint. The application can backoff and retry.

InvalidStateException - Can't store pending checkpoint. Unable to store the checkpoint in the DynamoDB table (e.g. table doesn't exist).

ThrottlingException - Can't store pending checkpoint. Can be caused by checkpointing too frequently. Consider increasing the throughput/capacity of the checkpoint store or reducing checkpoint frequency.

ShutdownException - The record processor instance has been shutdown. Another instance may have started processing some of these records already. The application should abort processing via this RecordProcessor instance.

prepareCheckpoint

public IPreparedCheckpointer prepareCheckpoint(String sequenceNumber,
                                               long subSequenceNumber)
                                        throws KinesisClientLibDependencyException,
                                               InvalidStateException,
                                               ThrottlingException,
                                               ShutdownException

This method will record a pending checkpoint at the provided sequenceNumber and subSequenceNumber, the latter for aggregated records produced with the Producer Library. This method is analogous to IRecordProcessorCheckpointer.prepareCheckpoint() but provides the ability to specify the sequence number at which to checkpoint

Specified by:

prepareCheckpoint in interface IRecordProcessorCheckpointer

Parameters:

sequenceNumber - A sequence number at which to prepare checkpoint in this shard.

subSequenceNumber - A subsequence number at which to prepare checkpoint within this shard.

Returns:

an IPreparedCheckpointer object that can be called later to persist the checkpoint.

Throws:

KinesisClientLibDependencyException - Encountered an issue when storing the pending checkpoint. The application can backoff and retry.

InvalidStateException - Can't store pending checkpoint. Unable to store the checkpoint in the DynamoDB table (e.g. table doesn't exist).

ThrottlingException - Can't store pending checkpoint. Can be caused by checkpointing too frequently. Consider increasing the throughput/capacity of the checkpoint store or reducing checkpoint frequency.

ShutdownException - The record processor instance has been shutdown. Another instance may have started processing some of these records already. The application should abort processing via this RecordProcessor instance.

getLastCheckpointValue

public ExtendedSequenceNumber getLastCheckpointValue()

Returns:

the lastCheckpointValue

getLargestPermittedCheckpointValue

public ExtendedSequenceNumber getLargestPermittedCheckpointValue()

Used for testing.

Returns:

the largest permitted checkpoint

setLargestPermittedCheckpointValue

public void setLargestPermittedCheckpointValue(ExtendedSequenceNumber largestPermittedCheckpointValue)

Parameters:

largestPermittedCheckpointValue - the largest permitted checkpoint

setSequenceNumberAtShardEnd

public void setSequenceNumberAtShardEnd(ExtendedSequenceNumber extendedSequenceNumber)

Used to remember the last extended sequence number before SHARD_END to allow us to prevent the checkpointer from checkpointing at the end of the shard twice (i.e. at the last extended sequence number and then again at SHARD_END).

Parameters:

extendedSequenceNumber -