software.amazon.kinesis.leases.dynamodb

Class DynamoDBLeaseCoordinator

java.lang.Object

software.amazon.kinesis.leases.dynamodb.DynamoDBLeaseCoordinator

All Implemented Interfaces:

LeaseCoordinator

public class DynamoDBLeaseCoordinator
extends Object
implements LeaseCoordinator

LeaseCoordinator abstracts away LeaseTaker and LeaseRenewer from the application code that's using leasing. It owns the scheduling of the two previously mentioned components as well as informing LeaseRenewer when LeaseTaker takes new leases.

Field Summary

Fields

Modifier and Type	Field and Description
protected MetricsFactory	metricsFactory

Constructor Summary

Constructors

Constructor and Description

DynamoDBLeaseCoordinator(LeaseRefresher leaseRefresher, String workerIdentifier, long leaseDurationMillis, boolean enablePriorityLeaseAssignment, long epsilonMillis, int maxLeasesForWorker, int maxLeasesToStealAtOneTime, int maxLeaseRenewerThreadCount, long initialLeaseTableReadCapacity, long initialLeaseTableWriteCapacity, MetricsFactory metricsFactory, LeaseManagementConfig.WorkerUtilizationAwareAssignmentConfig workerUtilizationAwareAssignmentConfig, LeaseManagementConfig.GracefulLeaseHandoffConfig gracefulLeaseHandoffConfig, ConcurrentMap<ShardInfo,ShardConsumer> shardInfoShardConsumerMap) Constructor.

Method Summary

Modifier and Type	Method and Description
List<Lease>	allLeases() Default implementation returns an empty list and concrete implementation is expected to return all leases for the application that are in the lease table.
static ShardInfo	convertLeaseToAssignment(Lease lease) Utility method to convert the basic lease or multistream lease to ShardInfo
void	dropLease(Lease lease) Requests that renewals for the given lease are stopped.
Collection<Lease>	getAssignments()
List<ShardInfo>	getCurrentAssignments()
Lease	getCurrentlyHeldLease(String leaseKey)
void	initialize() Initialize the lease coordinator (create the lease table if needed).
DynamoDBLeaseCoordinator	initialLeaseTableReadCapacity(long readCapacity) Deprecated.
DynamoDBLeaseCoordinator	initialLeaseTableWriteCapacity(long writeCapacity) Deprecated.
boolean	isRunning()
LeaseRefresher	leaseRefresher()
LeaseStatsRecorder	leaseStatsRecorder()
void	runLeaseRenewer() Runs a single iteration of the lease renewer - used by integration tests.
void	runLeaseTaker() Runs a single iteration of the lease taker - used by integration tests.
void	start(MigrationAdaptiveLeaseAssignmentModeProvider leaseAssignmentModeProvider) Start background LeaseHolder and LeaseTaker threads.
void	stop() Stops background threads and waits for specific amount of time for all background tasks to complete.
void	stopLeaseTaker() Requests the cancellation of the lease taker.
boolean	updateLease(Lease lease, UUID concurrencyToken, String operation, String singleStreamShardId) Updates application-specific lease values in DynamoDB.
String	workerIdentifier()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

metricsFactory

protected final MetricsFactory metricsFactory

Constructor Detail

DynamoDBLeaseCoordinator

public DynamoDBLeaseCoordinator(LeaseRefresher leaseRefresher,
                                String workerIdentifier,
                                long leaseDurationMillis,
                                boolean enablePriorityLeaseAssignment,
                                long epsilonMillis,
                                int maxLeasesForWorker,
                                int maxLeasesToStealAtOneTime,
                                int maxLeaseRenewerThreadCount,
                                long initialLeaseTableReadCapacity,
                                long initialLeaseTableWriteCapacity,
                                MetricsFactory metricsFactory,
                                LeaseManagementConfig.WorkerUtilizationAwareAssignmentConfig workerUtilizationAwareAssignmentConfig,
                                LeaseManagementConfig.GracefulLeaseHandoffConfig gracefulLeaseHandoffConfig,
                                ConcurrentMap<ShardInfo,ShardConsumer> shardInfoShardConsumerMap)

Constructor.

Parameters:

leaseRefresher - LeaseRefresher instance to use

workerIdentifier - Identifies the worker (e.g. useful to track lease ownership)

leaseDurationMillis - Duration of a lease

enablePriorityLeaseAssignment - Whether to enable priority lease assignment for very expired leases

epsilonMillis - Allow for some variance when calculating lease expirations

maxLeasesForWorker - Max leases this Worker can handle at a time

maxLeasesToStealAtOneTime - Steal up to these many leases at a time (for load balancing)

initialLeaseTableReadCapacity - Initial dynamodb lease table read iops if creating the lease table

initialLeaseTableWriteCapacity - Initial dynamodb lease table write iops if creating the lease table

metricsFactory - Used to publish metrics about lease operations

Method Detail

initialize

public void initialize()
                throws ProvisionedThroughputException,
                       DependencyException,
                       IllegalStateException

Description copied from interface: LeaseCoordinator

Initialize the lease coordinator (create the lease table if needed).

Specified by:

initialize in interface LeaseCoordinator

Throws:

ProvisionedThroughputException

DependencyException

IllegalStateException

start

public void start(MigrationAdaptiveLeaseAssignmentModeProvider leaseAssignmentModeProvider)
           throws DependencyException,
                  InvalidStateException,
                  ProvisionedThroughputException

Description copied from interface: LeaseCoordinator

Start background LeaseHolder and LeaseTaker threads.

Specified by:

start in interface LeaseCoordinator

Parameters:

leaseAssignmentModeProvider - provider of Lease Assignment mode to determine whether to start components for both V2 and V3 functionality or only V3 functionality

Throws:

DependencyException - If we encountered exception taking to DynamoDB

InvalidStateException - If the lease table doesn't exist

ProvisionedThroughputException - If we can't talk to DynamoDB due to insufficient capacity.

runLeaseTaker

public void runLeaseTaker()
                   throws DependencyException,
                          InvalidStateException

Description copied from interface: LeaseCoordinator

Runs a single iteration of the lease taker - used by integration tests.

Specified by:

runLeaseTaker in interface LeaseCoordinator

Throws:

DependencyException

InvalidStateException

runLeaseRenewer

public void runLeaseRenewer()
                     throws DependencyException,
                            InvalidStateException

Description copied from interface: LeaseCoordinator

Runs a single iteration of the lease renewer - used by integration tests.

Specified by:

runLeaseRenewer in interface LeaseCoordinator

Throws:

DependencyException

InvalidStateException

getAssignments

public Collection<Lease> getAssignments()

Specified by:

getAssignments in interface LeaseCoordinator

Returns:

currently held leases

allLeases

public List<Lease> allLeases()

Description copied from interface: LeaseCoordinator

Default implementation returns an empty list and concrete implementation is expected to return all leases for the application that are in the lease table. This enables application managing Kcl Scheduler to take care of horizontal scaling for example.

Specified by:

allLeases in interface LeaseCoordinator

Returns:

all leases for the application that are in the lease table

getCurrentlyHeldLease

public Lease getCurrentlyHeldLease(String leaseKey)

Specified by:

getCurrentlyHeldLease in interface LeaseCoordinator

Parameters:

leaseKey - lease key to fetch currently held lease for

Returns:

deep copy of currently held Lease for given key, or null if we don't hold the lease for that key

workerIdentifier

public String workerIdentifier()

Specified by:

workerIdentifier in interface LeaseCoordinator

Returns:

workerIdentifier

leaseRefresher

public LeaseRefresher leaseRefresher()

Specified by:

leaseRefresher in interface LeaseCoordinator

Returns:

LeaseRefresher

stop

public void stop()

Description copied from interface: LeaseCoordinator

Stops background threads and waits for specific amount of time for all background tasks to complete. If tasks are not completed after this time, method will shutdown thread pool forcefully and return.

Specified by:

stop in interface LeaseCoordinator

stopLeaseTaker

public void stopLeaseTaker()

Description copied from interface: LeaseCoordinator

Requests the cancellation of the lease taker.

Specified by:

stopLeaseTaker in interface LeaseCoordinator

dropLease

public void dropLease(Lease lease)

Description copied from interface: LeaseCoordinator

Requests that renewals for the given lease are stopped.

Specified by:

dropLease in interface LeaseCoordinator

Parameters:

lease - the lease to stop renewing.

isRunning

public boolean isRunning()

Specified by:

isRunning in interface LeaseCoordinator

Returns:

true if this LeaseCoordinator is running

updateLease

public boolean updateLease(Lease lease,
                           UUID concurrencyToken,
                           String operation,
                           String singleStreamShardId)
                    throws DependencyException,
                           InvalidStateException,
                           ProvisionedThroughputException

Description copied from interface: LeaseCoordinator

Updates application-specific lease values in DynamoDB.

Specified by:

updateLease in interface LeaseCoordinator

Parameters:

lease - lease object containing updated values

concurrencyToken - obtained by calling Lease.concurrencyToken for a currently held lease

operation - that performs updateLease

singleStreamShardId - for metrics emission in single stream mode. MultiStream mode will get the shardId from the lease object

Returns:

true if update succeeded, false otherwise

Throws:

DependencyException - if DynamoDB update fails in an unexpected way

InvalidStateException - if lease table does not exist

ProvisionedThroughputException - if DynamoDB update fails due to lack of capacity

getCurrentAssignments

public List<ShardInfo> getCurrentAssignments()

Specified by:

getCurrentAssignments in interface LeaseCoordinator

Returns:

Current shard/lease assignments

convertLeaseToAssignment

public static ShardInfo convertLeaseToAssignment(Lease lease)

Utility method to convert the basic lease or multistream lease to ShardInfo

Parameters:

lease -

Returns:

ShardInfo

initialLeaseTableReadCapacity

@Deprecated
public DynamoDBLeaseCoordinator initialLeaseTableReadCapacity(long readCapacity)

Deprecated.

NOTE: This method is deprecated. Please set the initial capacity through the constructor.

This is a method of the public lease coordinator interface.

Specified by:

initialLeaseTableReadCapacity in interface LeaseCoordinator

Parameters:

readCapacity - The DynamoDB table used for tracking leases will be provisioned with the specified initial read capacity

Returns:

LeaseCoordinator

initialLeaseTableWriteCapacity

@Deprecated
public DynamoDBLeaseCoordinator initialLeaseTableWriteCapacity(long writeCapacity)

Deprecated.

NOTE: This method is deprecated. Please set the initial capacity through the constructor.

This is a method of the public lease coordinator interface.

Specified by:

initialLeaseTableWriteCapacity in interface LeaseCoordinator

Parameters:

writeCapacity - The DynamoDB table used for tracking leases will be provisioned with the specified initial write capacity

Returns:

LeaseCoordinator

leaseStatsRecorder

public LeaseStatsRecorder leaseStatsRecorder()

Specified by:

leaseStatsRecorder in interface LeaseCoordinator

Returns:

instance of LeaseStatsRecorder