software.amazon.kinesis.leases.dynamodb

Class DynamoDBLeaseRefresher

java.lang.Object

software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseRefresher

All Implemented Interfaces:

LeaseRefresher

public class DynamoDBLeaseRefresher
extends Object
implements LeaseRefresher

An implementation of LeaseRefresher that uses DynamoDB.

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	consistentReads
protected software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient	dynamoDBClient
protected LeaseSerializer	serializer
protected String	table

Constructor Summary

Constructors

Constructor and Description

DynamoDBLeaseRefresher(String table, software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient dynamoDBClient, LeaseSerializer serializer, boolean consistentReads, @NonNull TableCreatorCallback tableCreatorCallback, Duration dynamoDbRequestTimeout, DdbTableConfig ddbTableConfig, boolean leaseTableDeletionProtectionEnabled, boolean leaseTablePitrEnabled, Collection<software.amazon.awssdk.services.dynamodb.model.Tag> tags) Constructor.

Method Summary

Modifier and Type	Method and Description
boolean	assignLease(Lease lease, String newOwner) Assigns given lease to newOwner owner by incrementing its leaseCounter and setting its owner field.
protected DependencyException	convertAndRethrowExceptions(String operation, String leaseKey, Exception e)
boolean	createLeaseIfNotExists(@NonNull Lease lease) Create a new lease.
String	createLeaseOwnerToLeaseKeyIndexIfNotExists() Creates the LeaseOwnerToLeaseKey index on the lease table if it doesn't exist and returns the status of index.
boolean	createLeaseTableIfNotExists() Creates the table that will store leases.
boolean	createLeaseTableIfNotExists(@NonNull Long readCapacity, @NonNull Long writeCapacity) Creates the table that will store leases.
void	deleteAll() Delete all leases from DynamoDB.
void	deleteLease(@NonNull Lease lease) Delete the given lease from DynamoDB.
boolean	evictLease(@NonNull Lease lease) Evict the current owner of lease by setting owner to null.
ExtendedSequenceNumber	getCheckpoint(String leaseKey) Gets the current checkpoint of the shard.
Lease	getLease(@NonNull String leaseKey)
String	getLeaseTableIdentifier()
boolean	initiateGracefulLeaseHandoff(Lease lease, String newOwner) Initiates a graceful handoff of the given lease to the specified new owner, allowing the current owner to complete its processing before transferring ownership.
boolean	isLeaseOwnerToLeaseKeyIndexActive() Check if leaseOwner GSI is ACTIVE
boolean	isLeaseTableEmpty() Check (synchronously) if there are any leases in the lease table.
boolean	leaseTableExists()
List<String>	listLeaseKeysForWorker(String workerIdentifier) List all leases for a given workerIdentifier synchronously.
List<Lease>	listLeases() List all objects in table synchronously.
List<Lease>	listLeasesForStream(StreamIdentifier streamIdentifier) List all leases for a given stream synchronously.
Map.Entry<List<Lease>,List<String>>	listLeasesParallely(ExecutorService parallelScanExecutorService, int parallelScanTotalSegment) List all leases from the storage parallely and deserialize into Lease objects.
boolean	renewLease(@NonNull Lease lease) Renew a lease by incrementing the lease counter.
boolean	takeLease(@NonNull Lease lease, @NonNull String owner) Take a lease for the given owner by incrementing its leaseCounter and setting its owner field.
boolean	updateLease(@NonNull Lease lease) Update application-specific fields of the given lease in DynamoDB.
void	updateLeaseWithMetaInfo(Lease lease, UpdateField updateField) Update application-specific fields of the given lease in DynamoDB.
boolean	waitUntilLeaseOwnerToLeaseKeyIndexExists(long secondsBetweenPolls, long timeoutSeconds) Blocks until the index exists by polling storage till either the index is ACTIVE or else timeout has happened.
boolean	waitUntilLeaseTableExists(long secondsBetweenPolls, long timeoutSeconds) Blocks until the lease table exists by polling leaseTableExists.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

table

protected final String table

dynamoDBClient

protected final software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient dynamoDBClient

serializer

protected final LeaseSerializer serializer

consistentReads

protected final boolean consistentReads

Constructor Detail

DynamoDBLeaseRefresher

public DynamoDBLeaseRefresher(String table,
                              software.amazon.awssdk.services.dynamodb.DynamoDbAsyncClient dynamoDBClient,
                              LeaseSerializer serializer,
                              boolean consistentReads,
                              @NonNull
                              @NonNull TableCreatorCallback tableCreatorCallback,
                              Duration dynamoDbRequestTimeout,
                              DdbTableConfig ddbTableConfig,
                              boolean leaseTableDeletionProtectionEnabled,
                              boolean leaseTablePitrEnabled,
                              Collection<software.amazon.awssdk.services.dynamodb.model.Tag> tags)

Constructor.

Parameters:

table -

dynamoDBClient -

serializer -

consistentReads -

tableCreatorCallback -

dynamoDbRequestTimeout -

ddbTableConfig -

leaseTableDeletionProtectionEnabled -

leaseTablePitrEnabled -

tags -

Method Detail

createLeaseTableIfNotExists

public boolean createLeaseTableIfNotExists(@NonNull
                                           @NonNull Long readCapacity,
                                           @NonNull
                                           @NonNull Long writeCapacity)
                                    throws ProvisionedThroughputException,
                                           DependencyException

Creates the table that will store leases. Succeeds if table already exists. Deprecated. Use LeaseRefresher.createLeaseTableIfNotExists(). This method always creates table in PROVISIONED mode and with RCU and WCU provided as method args

Specified by:

createLeaseTableIfNotExists in interface LeaseRefresher

Returns:

true if we created a new table (table didn't exist before)

Throws:

ProvisionedThroughputException - if we cannot create the lease table due to per-AWS-account capacity restrictions.

DependencyException - if DynamoDB createTable fails in an unexpected way

createLeaseTableIfNotExists

public boolean createLeaseTableIfNotExists()
                                    throws ProvisionedThroughputException,
                                           DependencyException

Creates the table that will store leases. Table is now created in PayPerRequest billing mode by default. Succeeds if table already exists.

Specified by:

createLeaseTableIfNotExists in interface LeaseRefresher

Returns:

true if we created a new table (table didn't exist before)

Throws:

ProvisionedThroughputException - if we cannot create the lease table due to per-AWS-account capacity restrictions.

DependencyException - if DynamoDB createTable fails in an unexpected way

leaseTableExists

public boolean leaseTableExists()
                         throws DependencyException

Specified by:

leaseTableExists in interface LeaseRefresher

Returns:

true if the lease table already exists.

Throws:

DependencyException - if DynamoDB describeTable fails in an unexpected way

waitUntilLeaseTableExists

public boolean waitUntilLeaseTableExists(long secondsBetweenPolls,
                                         long timeoutSeconds)
                                  throws DependencyException

Description copied from interface: LeaseRefresher

Blocks until the lease table exists by polling leaseTableExists.

Specified by:

waitUntilLeaseTableExists in interface LeaseRefresher

Parameters:

secondsBetweenPolls - time to wait between polls in seconds

timeoutSeconds - total time to wait in seconds

Returns:

true if table exists, false if timeout was reached

Throws:

DependencyException - if DynamoDB describeTable fails in an unexpected way

createLeaseOwnerToLeaseKeyIndexIfNotExists

public String createLeaseOwnerToLeaseKeyIndexIfNotExists()
                                                  throws DependencyException

Description copied from interface: LeaseRefresher

Creates the LeaseOwnerToLeaseKey index on the lease table if it doesn't exist and returns the status of index.

Specified by:

createLeaseOwnerToLeaseKeyIndexIfNotExists in interface LeaseRefresher

Returns:

indexStatus status of the index.

Throws:

DependencyException - if storage's describe API fails in an unexpected way

waitUntilLeaseOwnerToLeaseKeyIndexExists

public boolean waitUntilLeaseOwnerToLeaseKeyIndexExists(long secondsBetweenPolls,
                                                        long timeoutSeconds)

Description copied from interface: LeaseRefresher

Blocks until the index exists by polling storage till either the index is ACTIVE or else timeout has happened.

Specified by:

waitUntilLeaseOwnerToLeaseKeyIndexExists in interface LeaseRefresher

Parameters:

secondsBetweenPolls - time to wait between polls in seconds

timeoutSeconds - total time to wait in seconds

Returns:

true if index on the table exists and is ACTIVE, false if timeout was reached

isLeaseOwnerToLeaseKeyIndexActive

public boolean isLeaseOwnerToLeaseKeyIndexActive()
                                          throws DependencyException

Description copied from interface: LeaseRefresher

Check if leaseOwner GSI is ACTIVE

Specified by:

isLeaseOwnerToLeaseKeyIndexActive in interface LeaseRefresher

Returns:

true if index is active, false otherwise

Throws:

DependencyException - if storage's describe API fails in an unexpected way

listLeasesForStream

public List<Lease> listLeasesForStream(StreamIdentifier streamIdentifier)
                                throws DependencyException,
                                       InvalidStateException,
                                       ProvisionedThroughputException

List all leases for a given stream synchronously.

Specified by:

listLeasesForStream in interface LeaseRefresher

Returns:

list of leases

Throws:

DependencyException - if DynamoDB scan fails in an unexpected way

InvalidStateException - if lease table does not exist

ProvisionedThroughputException - if DynamoDB scan fails due to lack of capacity

listLeaseKeysForWorker

public List<String> listLeaseKeysForWorker(String workerIdentifier)
                                    throws DependencyException,
                                           InvalidStateException

List all leases for a given workerIdentifier synchronously. Default implementation calls listLeases() and filters the results. This method throws InvalidStateException in case of LEASE_OWNER_TO_LEASE_KEY_INDEX_NAME does not exists. If index creation is not done and want to listLeases for a worker, use listLeases() and filter on that to list leases.

Specified by:

listLeaseKeysForWorker in interface LeaseRefresher

Returns:

list of leases

Throws:

DependencyException - if DynamoDB scan fails in an unexpected way

InvalidStateException - if lease table does not exist

listLeases

public List<Lease> listLeases()
                       throws DependencyException,
                              InvalidStateException,
                              ProvisionedThroughputException

List all objects in table synchronously.

Specified by:

listLeases in interface LeaseRefresher

Returns:

list of leases

Throws:

DependencyException - if DynamoDB scan fails in an unexpected way

InvalidStateException - if lease table does not exist

ProvisionedThroughputException - if DynamoDB scan fails due to lack of capacity

listLeasesParallely

public Map.Entry<List<Lease>,List<String>> listLeasesParallely(ExecutorService parallelScanExecutorService,
                                                               int parallelScanTotalSegment)
                                                        throws DependencyException,
                                                               InvalidStateException,
                                                               ProvisionedThroughputException

Description copied from interface: LeaseRefresher

List all leases from the storage parallely and deserialize into Lease objects. Returns the list of leaseKey that failed deserialize separately.

Specified by:

listLeasesParallely in interface LeaseRefresher

Parameters:

parallelScanExecutorService - thread pool to use for parallel scan

parallelScanTotalSegment - no. of parallel scans. If parallelismFactor is 0 then parallelismFactor will be calculated based on table size

Returns:

Pair of List of leases from the storage and List of items failed to deserialize

Throws:

DependencyException - if DynamoDB scan fails in an unexpected way

InvalidStateException - if lease table does not exist

ProvisionedThroughputException - if DynamoDB scan fails due to lack of capacity

isLeaseTableEmpty

public boolean isLeaseTableEmpty()
                          throws DependencyException,
                                 InvalidStateException,
                                 ProvisionedThroughputException

Check (synchronously) if there are any leases in the lease table. Current implementation has a fixed 10 parallism

Specified by:

isLeaseTableEmpty in interface LeaseRefresher

Returns:

true if there are no leases in the lease table

Throws:

DependencyException - if DynamoDB scan fails in an unexpected way

InvalidStateException - if lease table does not exist

ProvisionedThroughputException - if DynamoDB scan fails due to lack of capacity

createLeaseIfNotExists

public boolean createLeaseIfNotExists(@NonNull
                                      @NonNull Lease lease)
                               throws DependencyException,
                                      InvalidStateException,
                                      ProvisionedThroughputException

Create a new lease. Conditional on a lease not already existing with this shardId.

Specified by:

createLeaseIfNotExists in interface LeaseRefresher

Parameters:

lease - the lease to create

Returns:

true if lease was created, false if lease already exists

Throws:

DependencyException - if DynamoDB put fails in an unexpected way

InvalidStateException - if lease table does not exist

ProvisionedThroughputException - if DynamoDB put fails due to lack of capacity

getLease

public Lease getLease(@NonNull
                      @NonNull String leaseKey)
               throws DependencyException,
                      InvalidStateException,
                      ProvisionedThroughputException

Specified by:

getLease in interface LeaseRefresher

Parameters:

leaseKey - Get the lease for this leasekey

Returns:

lease for the specified leaseKey, or null if one doesn't exist

Throws:

DependencyException - if DynamoDB get fails in an unexpected way

InvalidStateException - if lease table does not exist

ProvisionedThroughputException - if DynamoDB get fails due to lack of capacity

renewLease

public boolean renewLease(@NonNull
                          @NonNull Lease lease)
                   throws DependencyException,
                          InvalidStateException,
                          ProvisionedThroughputException

Renew a lease by incrementing the lease counter. Conditional on the leaseCounter in DynamoDB matching the leaseCounter of the input. Mutates the leaseCounter of the passed-in lease object after updating the record in DynamoDB.

Specified by:

renewLease in interface LeaseRefresher

Parameters:

lease - the lease to renew

Returns:

true if renewal succeeded, false otherwise

Throws:

DependencyException - if DynamoDB update fails in an unexpected way

InvalidStateException - if lease table does not exist

ProvisionedThroughputException - if DynamoDB update fails due to lack of capacity

takeLease

public boolean takeLease(@NonNull
                         @NonNull Lease lease,
                         @NonNull
                         @NonNull String owner)
                  throws DependencyException,
                         InvalidStateException,
                         ProvisionedThroughputException

Take a lease for the given owner by incrementing its leaseCounter and setting its owner field. Conditional on the leaseCounter in DynamoDB matching the leaseCounter of the input. Mutates the leaseCounter and owner of the passed-in lease object after updating DynamoDB.

Specified by:

takeLease in interface LeaseRefresher

Parameters:

lease - the lease to take

owner - the new owner

Returns:

true if lease was successfully taken, false otherwise

Throws:

DependencyException - if DynamoDB update fails in an unexpected way

InvalidStateException - if lease table does not exist

ProvisionedThroughputException - if DynamoDB update fails due to lack of capacity

initiateGracefulLeaseHandoff

public boolean initiateGracefulLeaseHandoff(Lease lease,
                                            String newOwner)
                                     throws DependencyException,
                                            InvalidStateException,
                                            ProvisionedThroughputException

Initiates a graceful handoff of the given lease to the specified new owner, allowing the current owner to complete its processing before transferring ownership.

This method updates the lease with the new owner information but ensures that the current owner is given time to gracefully finish its work (e.g., processing records) before the lease is reassigned.

Specified by:

initiateGracefulLeaseHandoff in interface LeaseRefresher

Parameters:

lease - the lease to be assigned

newOwner - the new owner

Returns:

true if a graceful handoff was successfully initiated

Throws:

DependencyException - if DynamoDB update fails in an unexpected way

InvalidStateException - if lease table does not exist

ProvisionedThroughputException - if DynamoDB update fails due to lack of capacity

assignLease

public boolean assignLease(Lease lease,
                           String newOwner)
                    throws DependencyException,
                           InvalidStateException,
                           ProvisionedThroughputException

Assigns given lease to newOwner owner by incrementing its leaseCounter and setting its owner field. Conditional on the leaseOwner in DynamoDB matching the leaseOwner of the input lease. Mutates the leaseCounter and owner of the passed-in lease object after updating DynamoDB.

Specified by:

assignLease in interface LeaseRefresher

Parameters:

lease - the lease to be assigned

newOwner - the new owner

Returns:

true if lease was successfully assigned, false otherwise

Throws:

DependencyException - if DynamoDB update fails in an unexpected way

InvalidStateException - if lease table does not exist

ProvisionedThroughputException - if DynamoDB update fails due to lack of capacity

evictLease

public boolean evictLease(@NonNull
                          @NonNull Lease lease)
                   throws DependencyException,
                          InvalidStateException,
                          ProvisionedThroughputException

Evict the current owner of lease by setting owner to null. Conditional on the owner in DynamoDB matching the owner of the input. Mutates the lease counter and owner of the passed-in lease object after updating the record in DynamoDB.

Specified by:

evictLease in interface LeaseRefresher

Parameters:

lease - the lease to void

Returns:

true if eviction succeeded, false otherwise

Throws:

DependencyException - if DynamoDB update fails in an unexpected way

InvalidStateException - if lease table does not exist

ProvisionedThroughputException - if DynamoDB update fails due to lack of capacity

deleteAll

public void deleteAll()
               throws DependencyException,
                      InvalidStateException,
                      ProvisionedThroughputException

Delete all leases from DynamoDB. Useful for tools/utils and testing.

Specified by:

deleteAll in interface LeaseRefresher

Throws:

DependencyException - if DynamoDB scan or delete fail in an unexpected way

InvalidStateException - if lease table does not exist

ProvisionedThroughputException - if DynamoDB scan or delete fail due to lack of capacity

deleteLease

public void deleteLease(@NonNull
                        @NonNull Lease lease)
                 throws DependencyException,
                        InvalidStateException,
                        ProvisionedThroughputException

Delete the given lease from DynamoDB. Does nothing when passed a lease that does not exist in DynamoDB.

Specified by:

deleteLease in interface LeaseRefresher

Parameters:

lease - the lease to delete

Throws:

DependencyException - if DynamoDB delete fails in an unexpected way

InvalidStateException - if lease table does not exist

ProvisionedThroughputException - if DynamoDB delete fails due to lack of capacity

updateLease

public boolean updateLease(@NonNull
                           @NonNull Lease lease)
                    throws DependencyException,
                           InvalidStateException,
                           ProvisionedThroughputException

Update application-specific fields of the given lease in DynamoDB. Does not update fields managed by the leasing library such as leaseCounter, leaseOwner, or leaseKey. Conditional on the leaseCounter in DynamoDB matching the leaseCounter of the input. Increments the lease counter in DynamoDB so that updates can be contingent on other updates. Mutates the lease counter of the passed-in lease object.

Specified by:

updateLease in interface LeaseRefresher

Returns:

true if update succeeded, false otherwise

Throws:

DependencyException - if DynamoDB update fails in an unexpected way

InvalidStateException - if lease table does not exist

ProvisionedThroughputException - if DynamoDB update fails due to lack of capacity

updateLeaseWithMetaInfo

public void updateLeaseWithMetaInfo(Lease lease,
                                    UpdateField updateField)
                             throws DependencyException,
                                    InvalidStateException,
                                    ProvisionedThroughputException

Description copied from interface: LeaseRefresher

Update application-specific fields of the given lease in DynamoDB. Does not update fields managed by the leasing library such as leaseCounter, leaseOwner, or leaseKey.

Specified by:

updateLeaseWithMetaInfo in interface LeaseRefresher

Throws:

DependencyException - if DynamoDB update fails in an unexpected way

InvalidStateException - if lease table does not exist

ProvisionedThroughputException - if DynamoDB update fails due to lack of capacity

getCheckpoint

public ExtendedSequenceNumber getCheckpoint(String leaseKey)
                                     throws ProvisionedThroughputException,
                                            InvalidStateException,
                                            DependencyException

Gets the current checkpoint of the shard. This is useful in the resharding use case where we will wait for the parent shard to complete before starting on the records from a child shard.

Specified by:

getCheckpoint in interface LeaseRefresher

Parameters:

leaseKey - Checkpoint of this shard will be returned

Returns:

Checkpoint of this shard, or null if the shard record doesn't exist.

Throws:

ProvisionedThroughputException - if DynamoDB update fails due to lack of capacity

InvalidStateException - if lease table does not exist

DependencyException - if DynamoDB update fails in an unexpected way

getLeaseTableIdentifier

public String getLeaseTableIdentifier()
                               throws DependencyException

Specified by:

getLeaseTableIdentifier in interface LeaseRefresher

Throws:

DependencyException

convertAndRethrowExceptions

protected DependencyException convertAndRethrowExceptions(String operation,
                                                          String leaseKey,
                                                          Exception e)
                                                   throws ProvisionedThroughputException,
                                                          InvalidStateException

Throws:

ProvisionedThroughputException

InvalidStateException