com.amazonaws.services.kinesis.clientlibrary.lib.worker

Class Worker

java.lang.Object

com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker

All Implemented Interfaces:

Runnable

public class Worker
extends Object
implements Runnable

Worker is the high level class that Kinesis applications use to start processing data. It initializes and oversees different components (e.g. syncing shard and lease information, tracking shard assignments, and processing data from the shards).

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Worker.Builder Builder to construct a Worker instance.

Constructor Summary

Constructors

Constructor and Description
Worker(IRecordProcessorFactory recordProcessorFactory, KinesisClientLibConfiguration config) Constructor.
Worker(IRecordProcessorFactory recordProcessorFactory, KinesisClientLibConfiguration config, com.amazonaws.services.kinesis.AmazonKinesis kinesisClient, com.amazonaws.services.dynamodbv2.AmazonDynamoDB dynamoDBClient, com.amazonaws.services.cloudwatch.AmazonCloudWatch cloudWatchClient)
Worker(IRecordProcessorFactory recordProcessorFactory, KinesisClientLibConfiguration config, com.amazonaws.services.kinesis.AmazonKinesis kinesisClient, com.amazonaws.services.dynamodbv2.AmazonDynamoDB dynamoDBClient, com.amazonaws.services.cloudwatch.AmazonCloudWatch cloudWatchClient, ExecutorService execService)
Worker(IRecordProcessorFactory recordProcessorFactory, KinesisClientLibConfiguration config, com.amazonaws.services.kinesis.AmazonKinesis kinesisClient, com.amazonaws.services.dynamodbv2.AmazonDynamoDB dynamoDBClient, IMetricsFactory metricsFactory, ExecutorService execService)
Worker(IRecordProcessorFactory recordProcessorFactory, KinesisClientLibConfiguration config, com.amazonaws.services.kinesis.AmazonKinesisClient kinesisClient, com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient dynamoDBClient, com.amazonaws.services.cloudwatch.AmazonCloudWatchClient cloudWatchClient) This constructor is for binary compatibility with code compiled against version of the KCL that only have constructors taking "Client" objects.
Worker(IRecordProcessorFactory recordProcessorFactory, KinesisClientLibConfiguration config, com.amazonaws.services.kinesis.AmazonKinesisClient kinesisClient, com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient dynamoDBClient, com.amazonaws.services.cloudwatch.AmazonCloudWatchClient cloudWatchClient, ExecutorService execService) This constructor is for binary compatibility with code compiled against version of the KCL that only have constructors taking "Client" objects.
Worker(IRecordProcessorFactory recordProcessorFactory, KinesisClientLibConfiguration config, com.amazonaws.services.kinesis.AmazonKinesisClient kinesisClient, com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient dynamoDBClient, IMetricsFactory metricsFactory, ExecutorService execService) This constructor is for binary compatibility with code compiled against version of the KCL that only have constructors taking "Client" objects.
Worker(IRecordProcessorFactory recordProcessorFactory, KinesisClientLibConfiguration config, ExecutorService execService) Constructor.
Worker(IRecordProcessorFactory recordProcessorFactory, KinesisClientLibConfiguration config, IMetricsFactory metricsFactory)
Worker(IRecordProcessorFactory recordProcessorFactory, KinesisClientLibConfiguration config, IMetricsFactory metricsFactory, ExecutorService execService)

Method Summary

Modifier and Type	Method and Description
protected com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardConsumer	buildConsumer(ShardInfo shardInfo, IRecordProcessorFactory factory)
String	getApplicationName()
Future<Void>	requestShutdown() Requests shutdown of the worker, notifying record processors, that implement IShutdownNotificationAware, of the impending shutdown.
void	run() Start consuming data from the stream, and pass it to the application record processors.
void	shutdown() Signals worker to shutdown.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Worker

public Worker(IRecordProcessorFactory recordProcessorFactory,
              KinesisClientLibConfiguration config)

Constructor.

Parameters:

recordProcessorFactory - Used to get record processor instances for processing data from shards

config - Kinesis Client Library configuration

Worker

public Worker(IRecordProcessorFactory recordProcessorFactory,
              KinesisClientLibConfiguration config,
              ExecutorService execService)

Constructor.

Parameters:

recordProcessorFactory - Used to get record processor instances for processing data from shards

config - Kinesis Client Library configuration

execService - ExecutorService to use for processing records (support for multi-threaded consumption)

Worker

public Worker(IRecordProcessorFactory recordProcessorFactory,
              KinesisClientLibConfiguration config,
              IMetricsFactory metricsFactory)

Parameters:

recordProcessorFactory - Used to get record processor instances for processing data from shards

config - Kinesis Client Library configuration

metricsFactory - Metrics factory used to emit metrics

Worker

public Worker(IRecordProcessorFactory recordProcessorFactory,
              KinesisClientLibConfiguration config,
              IMetricsFactory metricsFactory,
              ExecutorService execService)

Parameters:

recordProcessorFactory - Used to get record processor instances for processing data from shards

config - Kinesis Client Library configuration

metricsFactory - Metrics factory used to emit metrics

execService - ExecutorService to use for processing records (support for multi-threaded consumption)

Worker

public Worker(IRecordProcessorFactory recordProcessorFactory,
              KinesisClientLibConfiguration config,
              com.amazonaws.services.kinesis.AmazonKinesis kinesisClient,
              com.amazonaws.services.dynamodbv2.AmazonDynamoDB dynamoDBClient,
              com.amazonaws.services.cloudwatch.AmazonCloudWatch cloudWatchClient)

Parameters:

recordProcessorFactory - Used to get record processor instances for processing data from shards

config - Kinesis Client Library configuration

kinesisClient - Kinesis Client used for fetching data

dynamoDBClient - DynamoDB client used for checkpoints and tracking leases

cloudWatchClient - CloudWatch Client for publishing metrics

Worker

public Worker(IRecordProcessorFactory recordProcessorFactory,
              KinesisClientLibConfiguration config,
              com.amazonaws.services.kinesis.AmazonKinesis kinesisClient,
              com.amazonaws.services.dynamodbv2.AmazonDynamoDB dynamoDBClient,
              com.amazonaws.services.cloudwatch.AmazonCloudWatch cloudWatchClient,
              ExecutorService execService)

Parameters:

recordProcessorFactory - Used to get record processor instances for processing data from shards

config - Kinesis Client Library configuration

kinesisClient - Kinesis Client used for fetching data

dynamoDBClient - DynamoDB client used for checkpoints and tracking leases

cloudWatchClient - CloudWatch Client for publishing metrics

execService - ExecutorService to use for processing records (support for multi-threaded consumption)

Worker

public Worker(IRecordProcessorFactory recordProcessorFactory,
              KinesisClientLibConfiguration config,
              com.amazonaws.services.kinesis.AmazonKinesis kinesisClient,
              com.amazonaws.services.dynamodbv2.AmazonDynamoDB dynamoDBClient,
              IMetricsFactory metricsFactory,
              ExecutorService execService)

Parameters:

recordProcessorFactory - Used to get record processor instances for processing data from shards

config - Kinesis Client Library configuration

kinesisClient - Kinesis Client used for fetching data

dynamoDBClient - DynamoDB client used for checkpoints and tracking leases

metricsFactory - Metrics factory used to emit metrics

execService - ExecutorService to use for processing records (support for multi-threaded consumption)

Worker

public Worker(IRecordProcessorFactory recordProcessorFactory,
              KinesisClientLibConfiguration config,
              com.amazonaws.services.kinesis.AmazonKinesisClient kinesisClient,
              com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient dynamoDBClient,
              com.amazonaws.services.cloudwatch.AmazonCloudWatchClient cloudWatchClient)

This constructor is for binary compatibility with code compiled against version of the KCL that only have constructors taking "Client" objects.

Parameters:

recordProcessorFactory - Used to get record processor instances for processing data from shards

config - Kinesis Client Library configuration

kinesisClient - Kinesis Client used for fetching data

dynamoDBClient - DynamoDB client used for checkpoints and tracking leases

cloudWatchClient - CloudWatch Client for publishing metrics

Worker

public Worker(IRecordProcessorFactory recordProcessorFactory,
              KinesisClientLibConfiguration config,
              com.amazonaws.services.kinesis.AmazonKinesisClient kinesisClient,
              com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient dynamoDBClient,
              com.amazonaws.services.cloudwatch.AmazonCloudWatchClient cloudWatchClient,
              ExecutorService execService)

This constructor is for binary compatibility with code compiled against version of the KCL that only have constructors taking "Client" objects.

Parameters:

recordProcessorFactory - Used to get record processor instances for processing data from shards

config - Kinesis Client Library configuration

kinesisClient - Kinesis Client used for fetching data

dynamoDBClient - DynamoDB client used for checkpoints and tracking leases

cloudWatchClient - CloudWatch Client for publishing metrics

execService - ExecutorService to use for processing records (support for multi-threaded consumption)

Worker

public Worker(IRecordProcessorFactory recordProcessorFactory,
              KinesisClientLibConfiguration config,
              com.amazonaws.services.kinesis.AmazonKinesisClient kinesisClient,
              com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient dynamoDBClient,
              IMetricsFactory metricsFactory,
              ExecutorService execService)

This constructor is for binary compatibility with code compiled against version of the KCL that only have constructors taking "Client" objects.

Parameters:

recordProcessorFactory - Used to get record processor instances for processing data from shards

config - Kinesis Client Library configuration

kinesisClient - Kinesis Client used for fetching data

dynamoDBClient - DynamoDB client used for checkpoints and tracking leases

metricsFactory - Metrics factory used to emit metrics

execService - ExecutorService to use for processing records (support for multi-threaded consumption)

Method Detail

getApplicationName

public String getApplicationName()

Returns:

the applicationName

run

public void run()

Start consuming data from the stream, and pass it to the application record processors.

Specified by:

run in interface Runnable

requestShutdown

public Future<Void> requestShutdown()

Requests shutdown of the worker, notifying record processors, that implement IShutdownNotificationAware, of the impending shutdown. This gives the record processor a final chance to checkpoint. It's possible that a record processor won't be notify before being shutdown. This can occur if the lease is lost after requesting shutdown, but before the notification is dispatched.

Requested Shutdown Process

When a shutdown process is requested it operates slightly differently to allow the record processors a chance to checkpoint a final time.

1. Call to request shutdown invoked.
2. Worker stops attempting to acquire new leases
3. Record Processor Shutdown Begins
   1. Record processor is notified of the impending shutdown, and given a final chance to checkpoint
   2. The lease for the record processor is then dropped.
   3. The record processor enters into an idle state waiting for the worker to complete final termination
   4. The worker will detect a record processor that has lost it's lease, and will terminate the record processor with ShutdownReason.ZOMBIE
4. The worker will shutdown all record processors.
5. Once all record processors have been terminated, the worker will terminate all owned resources.
6. Once the worker shutdown is complete, the returned future is completed.

Returns:

a Future that will be set once the shutdown is complete.

shutdown

public void shutdown()

Signals worker to shutdown. Worker will try initiating shutdown of all record processors. Note that if executor services were passed to the worker by the user, worker will not attempt to shutdown those resources.

Shutdown Process

When called this will start shutdown of the record processor, and eventually shutdown the worker itself.

1. Call to start shutdown invoked
2. Lease coordinator told to stop taking leases, and to drop existing leases.
3. Worker discovers record processors that no longer have leases.
4. Worker triggers shutdown with state ShutdownReason.ZOMBIE.
5. Once all record processors are shutdown, worker terminates owned resources.
6. Shutdown complete.

buildConsumer

protected com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardConsumer buildConsumer(ShardInfo shardInfo,
                                                                                              IRecordProcessorFactory factory)