com.amazonaws.services.kinesis.clientlibrary.lib.worker

Class Worker

java.lang.Object

com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker

All Implemented Interfaces:

Runnable

public class Worker
extends Object
implements Runnable

Worker is the high level class that Kinesis applications use to start processing data. It initializes and oversees different components (e.g. syncing shard and lease information, tracking shard assignments, and processing data from the shards).

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Worker.Builder Builder to construct a Worker instance.

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	gracefuleShutdownStarted
protected com.amazonaws.services.kinesis.clientlibrary.lib.worker.GracefulShutdownCoordinator	gracefulShutdownCoordinator

Constructor Summary

Constructors

Constructor and Description
Worker(IRecordProcessorFactory recordProcessorFactory, KinesisClientLibConfiguration config) Constructor.
Worker(IRecordProcessorFactory recordProcessorFactory, KinesisClientLibConfiguration config, com.amazonaws.services.kinesis.AmazonKinesis kinesisClient, com.amazonaws.services.dynamodbv2.AmazonDynamoDB dynamoDBClient, com.amazonaws.services.cloudwatch.AmazonCloudWatch cloudWatchClient)
Worker(IRecordProcessorFactory recordProcessorFactory, KinesisClientLibConfiguration config, com.amazonaws.services.kinesis.AmazonKinesis kinesisClient, com.amazonaws.services.dynamodbv2.AmazonDynamoDB dynamoDBClient, com.amazonaws.services.cloudwatch.AmazonCloudWatch cloudWatchClient, ExecutorService execService)
Worker(IRecordProcessorFactory recordProcessorFactory, KinesisClientLibConfiguration config, com.amazonaws.services.kinesis.AmazonKinesis kinesisClient, com.amazonaws.services.dynamodbv2.AmazonDynamoDB dynamoDBClient, IMetricsFactory metricsFactory, ExecutorService execService)
Worker(IRecordProcessorFactory recordProcessorFactory, KinesisClientLibConfiguration config, com.amazonaws.services.kinesis.AmazonKinesisClient kinesisClient, com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient dynamoDBClient, com.amazonaws.services.cloudwatch.AmazonCloudWatchClient cloudWatchClient) This constructor is for binary compatibility with code compiled against version of the KCL that only have constructors taking "Client" objects.
Worker(IRecordProcessorFactory recordProcessorFactory, KinesisClientLibConfiguration config, com.amazonaws.services.kinesis.AmazonKinesisClient kinesisClient, com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient dynamoDBClient, com.amazonaws.services.cloudwatch.AmazonCloudWatchClient cloudWatchClient, ExecutorService execService) This constructor is for binary compatibility with code compiled against version of the KCL that only have constructors taking "Client" objects.
Worker(IRecordProcessorFactory recordProcessorFactory, KinesisClientLibConfiguration config, com.amazonaws.services.kinesis.AmazonKinesisClient kinesisClient, com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient dynamoDBClient, IMetricsFactory metricsFactory, ExecutorService execService) This constructor is for binary compatibility with code compiled against version of the KCL that only have constructors taking "Client" objects.
Worker(IRecordProcessorFactory recordProcessorFactory, KinesisClientLibConfiguration config, ExecutorService execService) Constructor.
Worker(IRecordProcessorFactory recordProcessorFactory, KinesisClientLibConfiguration config, IMetricsFactory metricsFactory)
Worker(IRecordProcessorFactory recordProcessorFactory, KinesisClientLibConfiguration config, IMetricsFactory metricsFactory, ExecutorService execService)

Method Summary

Modifier and Type	Method and Description
protected com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardConsumer	buildConsumer(ShardInfo shardInfo, IRecordProcessorFactory processorFactory)
Callable<Boolean>	createGracefulShutdownCallable() Creates a callable that will execute the graceful shutdown process.
String	getApplicationName()
boolean	hasGracefulShutdownStarted()
Future<Void>	requestShutdown() Deprecated.
void	run() Start consuming data from the stream, and pass it to the application record processors.
void	shutdown() Signals worker to shutdown.
Future<Boolean>	startGracefulShutdown() Requests a graceful shutdown of the worker, notifying record processors, that implement IShutdownNotificationAware, of the impending shutdown.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

gracefuleShutdownStarted

protected boolean gracefuleShutdownStarted

gracefulShutdownCoordinator

protected com.amazonaws.services.kinesis.clientlibrary.lib.worker.GracefulShutdownCoordinator gracefulShutdownCoordinator

Constructor Detail

Worker

public Worker(IRecordProcessorFactory recordProcessorFactory,
              KinesisClientLibConfiguration config)

Constructor.

Parameters:

recordProcessorFactory - Used to get record processor instances for processing data from shards

config - Kinesis Client Library configuration

Worker

public Worker(IRecordProcessorFactory recordProcessorFactory,
              KinesisClientLibConfiguration config,
              ExecutorService execService)

Constructor.

Parameters:

recordProcessorFactory - Used to get record processor instances for processing data from shards

config - Kinesis Client Library configuration

execService - ExecutorService to use for processing records (support for multi-threaded consumption)

Worker

public Worker(IRecordProcessorFactory recordProcessorFactory,
              KinesisClientLibConfiguration config,
              IMetricsFactory metricsFactory)

Parameters:

recordProcessorFactory - Used to get record processor instances for processing data from shards

config - Kinesis Client Library configuration

metricsFactory - Metrics factory used to emit metrics

Worker

public Worker(IRecordProcessorFactory recordProcessorFactory,
              KinesisClientLibConfiguration config,
              IMetricsFactory metricsFactory,
              ExecutorService execService)

Parameters:

recordProcessorFactory - Used to get record processor instances for processing data from shards

config - Kinesis Client Library configuration

metricsFactory - Metrics factory used to emit metrics

execService - ExecutorService to use for processing records (support for multi-threaded consumption)

Worker

public Worker(IRecordProcessorFactory recordProcessorFactory,
              KinesisClientLibConfiguration config,
              com.amazonaws.services.kinesis.AmazonKinesis kinesisClient,
              com.amazonaws.services.dynamodbv2.AmazonDynamoDB dynamoDBClient,
              com.amazonaws.services.cloudwatch.AmazonCloudWatch cloudWatchClient)

Parameters:

recordProcessorFactory - Used to get record processor instances for processing data from shards

config - Kinesis Client Library configuration

kinesisClient - Kinesis Client used for fetching data

dynamoDBClient - DynamoDB client used for checkpoints and tracking leases

cloudWatchClient - CloudWatch Client for publishing metrics

Worker

public Worker(IRecordProcessorFactory recordProcessorFactory,
              KinesisClientLibConfiguration config,
              com.amazonaws.services.kinesis.AmazonKinesis kinesisClient,
              com.amazonaws.services.dynamodbv2.AmazonDynamoDB dynamoDBClient,
              com.amazonaws.services.cloudwatch.AmazonCloudWatch cloudWatchClient,
              ExecutorService execService)

Parameters:

recordProcessorFactory - Used to get record processor instances for processing data from shards

config - Kinesis Client Library configuration

kinesisClient - Kinesis Client used for fetching data

dynamoDBClient - DynamoDB client used for checkpoints and tracking leases

cloudWatchClient - CloudWatch Client for publishing metrics

execService - ExecutorService to use for processing records (support for multi-threaded consumption)

Worker

public Worker(IRecordProcessorFactory recordProcessorFactory,
              KinesisClientLibConfiguration config,
              com.amazonaws.services.kinesis.AmazonKinesis kinesisClient,
              com.amazonaws.services.dynamodbv2.AmazonDynamoDB dynamoDBClient,
              IMetricsFactory metricsFactory,
              ExecutorService execService)

Parameters:

recordProcessorFactory - Used to get record processor instances for processing data from shards

config - Kinesis Client Library configuration

kinesisClient - Kinesis Client used for fetching data

dynamoDBClient - DynamoDB client used for checkpoints and tracking leases

metricsFactory - Metrics factory used to emit metrics

execService - ExecutorService to use for processing records (support for multi-threaded consumption)

Worker

public Worker(IRecordProcessorFactory recordProcessorFactory,
              KinesisClientLibConfiguration config,
              com.amazonaws.services.kinesis.AmazonKinesisClient kinesisClient,
              com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient dynamoDBClient,
              com.amazonaws.services.cloudwatch.AmazonCloudWatchClient cloudWatchClient)

This constructor is for binary compatibility with code compiled against version of the KCL that only have constructors taking "Client" objects.

Parameters:

recordProcessorFactory - Used to get record processor instances for processing data from shards

config - Kinesis Client Library configuration

kinesisClient - Kinesis Client used for fetching data

dynamoDBClient - DynamoDB client used for checkpoints and tracking leases

cloudWatchClient - CloudWatch Client for publishing metrics

Worker

public Worker(IRecordProcessorFactory recordProcessorFactory,
              KinesisClientLibConfiguration config,
              com.amazonaws.services.kinesis.AmazonKinesisClient kinesisClient,
              com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient dynamoDBClient,
              com.amazonaws.services.cloudwatch.AmazonCloudWatchClient cloudWatchClient,
              ExecutorService execService)

This constructor is for binary compatibility with code compiled against version of the KCL that only have constructors taking "Client" objects.

Parameters:

recordProcessorFactory - Used to get record processor instances for processing data from shards

config - Kinesis Client Library configuration

kinesisClient - Kinesis Client used for fetching data

dynamoDBClient - DynamoDB client used for checkpoints and tracking leases

cloudWatchClient - CloudWatch Client for publishing metrics

execService - ExecutorService to use for processing records (support for multi-threaded consumption)

Worker

public Worker(IRecordProcessorFactory recordProcessorFactory,
              KinesisClientLibConfiguration config,
              com.amazonaws.services.kinesis.AmazonKinesisClient kinesisClient,
              com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient dynamoDBClient,
              IMetricsFactory metricsFactory,
              ExecutorService execService)

This constructor is for binary compatibility with code compiled against version of the KCL that only have constructors taking "Client" objects.

Parameters:

recordProcessorFactory - Used to get record processor instances for processing data from shards

config - Kinesis Client Library configuration

kinesisClient - Kinesis Client used for fetching data

dynamoDBClient - DynamoDB client used for checkpoints and tracking leases

metricsFactory - Metrics factory used to emit metrics

execService - ExecutorService to use for processing records (support for multi-threaded consumption)

Method Detail

getApplicationName

public String getApplicationName()

Returns:

the applicationName

run

public void run()

Start consuming data from the stream, and pass it to the application record processors.

Specified by:

run in interface Runnable

requestShutdown

@Deprecated
public Future<Void> requestShutdown()

Deprecated.

Starts the requestedShutdown process, and returns a future that can be used to track the process. This is deprecated in favor of startGracefulShutdown(), which returns a more complete future, and indicates the process behavior

Returns:

a future that will be set once shutdown is completed.

startGracefulShutdown

public Future<Boolean> startGracefulShutdown()

Requests a graceful shutdown of the worker, notifying record processors, that implement IShutdownNotificationAware, of the impending shutdown. This gives the record processor a final chance to checkpoint. This will only create a single shutdown future. Additional attempts to start a graceful shutdown will return the previous future. It's possible that a record processor won't be notify before being shutdown. This can occur if the lease is lost after requesting shutdown, but before the notification is dispatched.

Requested Shutdown Process

When a shutdown process is requested it operates slightly differently to allow the record processors a chance to checkpoint a final time.

1. Call to request shutdown invoked.
2. Worker stops attempting to acquire new leases
3. Record Processor Shutdown Begins
   1. Record processor is notified of the impending shutdown, and given a final chance to checkpoint
   2. The lease for the record processor is then dropped.
   3. The record processor enters into an idle state waiting for the worker to complete final termination
   4. The worker will detect a record processor that has lost it's lease, and will terminate the record processor with ShutdownReason.ZOMBIE
4. The worker will shutdown all record processors.
5. Once all record processors have been terminated, the worker will terminate all owned resources.
6. Once the worker shutdown is complete, the returned future is completed.

Returns:

a future that will be set once the shutdown has completed. True indicates that the graceful shutdown completed successfully. A false value indicates that a non-exception case caused the shutdown process to terminate early.

createGracefulShutdownCallable

public Callable<Boolean> createGracefulShutdownCallable()

Creates a callable that will execute the graceful shutdown process. This callable can be used to execute graceful shutdowns in your own executor, or execute the shutdown synchronously.

Returns:

a callable that run the graceful shutdown process. This may return a callable that return true if the graceful shutdown has already been completed.

Throws:

IllegalStateException - thrown by the callable if another callable has already started the shutdown process.

hasGracefulShutdownStarted

public boolean hasGracefulShutdownStarted()

shutdown

public void shutdown()

Signals worker to shutdown. Worker will try initiating shutdown of all record processors. Note that if executor services were passed to the worker by the user, worker will not attempt to shutdown those resources.

Shutdown Process

When called this will start shutdown of the record processor, and eventually shutdown the worker itself.

1. Call to start shutdown invoked
2. Lease coordinator told to stop taking leases, and to drop existing leases.
3. Worker discovers record processors that no longer have leases.
4. Worker triggers shutdown with state ShutdownReason.ZOMBIE.
5. Once all record processors are shutdown, worker terminates owned resources.
6. Shutdown complete.

buildConsumer

protected com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardConsumer buildConsumer(ShardInfo shardInfo,
                                                                                              IRecordProcessorFactory processorFactory)