ai.djl.serving.wlm

Class PermanentBatchAggregator

java.lang.Object

ai.djl.serving.wlm.PermanentBatchAggregator

public class PermanentBatchAggregator
extends java.lang.Object

a batch aggregator that never terminates by itself. the surrounding thread has to be interrupted by sending an interrupt signal.

Field Summary

Fields

Modifier and Type	Field and Description
protected int	batchSize
protected java.util.concurrent.LinkedBlockingDeque<Job>	jobQueue
protected java.util.List<Job>	jobs

Constructor Summary

Constructors

Constructor and Description
PermanentBatchAggregator(ModelInfo model, java.util.concurrent.LinkedBlockingDeque<Job> jobQueue) Constructs a PermanentBatchAggregator instance.

Method Summary

Modifier and Type	Method and Description
java.util.List<ai.djl.modality.Input>	getRequest() Poll the queue and return a list of Input Objects for the model.
boolean	isFinished() Checks if this BatchAggregator and the thread can be shutdown or if this aggregator waits for more data.
protected java.util.List<Job>	pollBatch() Fills in the list with a batch of jobs.
void	sendError() Sends an internal server error.
void	sendResponse(java.util.List<ai.djl.modality.Output> outputs) Sends to response to all waiting clients.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

batchSize

protected int batchSize

jobs

protected java.util.List<Job> jobs

jobQueue

protected java.util.concurrent.LinkedBlockingDeque<Job> jobQueue

Constructor Detail

PermanentBatchAggregator

public PermanentBatchAggregator(ModelInfo model,
                                java.util.concurrent.LinkedBlockingDeque<Job> jobQueue)

Constructs a PermanentBatchAggregator instance.

Parameters:

model - the model to use.

jobQueue - the job queue for polling data from.

Method Detail

pollBatch

protected java.util.List<Job> pollBatch()
                                 throws java.lang.InterruptedException

Fills in the list with a batch of jobs.

Returns:

a list of jobs read by this batch interation.

Throws:

java.lang.InterruptedException - if interrupted

isFinished

public boolean isFinished()

Checks if this BatchAggregator and the thread can be shutdown or if this aggregator waits for more data.

Returns:

true if we can shutdown the thread. for example when max idle time exceeded in temporary batch aggregator.

getRequest

public java.util.List<ai.djl.modality.Input> getRequest()
                                                 throws java.lang.InterruptedException

Poll the queue and return a list of Input Objects for the model.

Returns:

list of input objects to pass to the model.

Throws:

java.lang.InterruptedException - if thread gets interrupted while waiting for new data in the queue.

sendResponse

public void sendResponse(java.util.List<ai.djl.modality.Output> outputs)

Sends to response to all waiting clients.

Parameters:

outputs - list of model-outputs in same order as the input objects.

sendError

public void sendError()

Sends an internal server error.