ai.djl.serving.wlm

Class TemporaryBatchAggregator

java.lang.Object

ai.djl.serving.wlm.TemporaryBatchAggregator

public class TemporaryBatchAggregator
extends java.lang.Object

a batch aggregator that terminates after a maximum idle time.

Field Summary

Fields

Modifier and Type	Field and Description
protected int	batchSize
protected java.util.concurrent.LinkedBlockingDeque<Job>	jobQueue
protected java.util.List<Job>	jobs

Constructor Summary

Constructors

Constructor and Description
TemporaryBatchAggregator(ModelInfo model, java.util.concurrent.LinkedBlockingDeque<Job> jobQueue) a batch aggregator that terminates after a maximum idle time.

Method Summary

Modifier and Type	Method and Description
java.util.List<ai.djl.modality.Input>	getRequest() Poll the queue and return a list of Input Objects for the model.
boolean	isFinished() Checks if this BatchAggregator and the thread can be shutdown or if this aggregator waits for more data.
protected java.util.List<Job>	pollBatch() Fills in the list with a batch of jobs.
void	sendError() Sends an internal server error.
void	sendResponse(java.util.List<ai.djl.modality.Output> outputs) Sends to response to all waiting clients.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

batchSize

protected int batchSize

jobs

protected java.util.List<Job> jobs

jobQueue

protected java.util.concurrent.LinkedBlockingDeque<Job> jobQueue

Constructor Detail

TemporaryBatchAggregator

public TemporaryBatchAggregator(ModelInfo model,
                                java.util.concurrent.LinkedBlockingDeque<Job> jobQueue)

a batch aggregator that terminates after a maximum idle time.

Parameters:

model - the model to run for.

jobQueue - reference to external job queue for polling.

Method Detail

pollBatch

protected java.util.List<Job> pollBatch()
                                 throws java.lang.InterruptedException

Fills in the list with a batch of jobs.

Returns:

a list of jobs read by this batch interation.

Throws:

java.lang.InterruptedException - if interrupted

isFinished

public boolean isFinished()

Checks if this BatchAggregator and the thread can be shutdown or if this aggregator waits for more data.

Returns:

true if we can shutdown the thread. for example when max idle time exceeded in temporary batch aggregator.

getRequest

public java.util.List<ai.djl.modality.Input> getRequest()
                                                 throws java.lang.InterruptedException

Poll the queue and return a list of Input Objects for the model.

Returns:

list of input objects to pass to the model.

Throws:

java.lang.InterruptedException - if thread gets interrupted while waiting for new data in the queue.

sendResponse

public void sendResponse(java.util.List<ai.djl.modality.Output> outputs)

Sends to response to all waiting clients.

Parameters:

outputs - list of model-outputs in same order as the input objects.

sendError

public void sendError()

Sends an internal server error.