ai.djl.serving.wlm

Class ModelManager

java.lang.Object

ai.djl.serving.wlm.ModelManager

public final class ModelManager
extends java.lang.Object

A class that in charge of managing models.

Method Summary

Modifier and Type	Method and Description
boolean	addJob(Job job) Adds an inference job to the job queue.
DescribeModelResponse	describeModel(java.lang.String modelName) Returns a list of worker information for specified model.
static ModelManager	getInstance() Returns the singleton ModelManager instance.
java.util.Map<java.lang.String,ModelInfo>	getModels() Returns the registry of all models.
java.util.Set<java.lang.String>	getStartupModels() Returns a set of models that was loaded at startup.
static void	init(ConfigManager configManager) Initialized the global ModelManager instance.
java.util.concurrent.CompletableFuture<ModelInfo>	registerModel(java.lang.String modelName, java.lang.String modelUrl, int batchSize, int maxBatchDelay, int maxIdleTime) Registers and loads a model.
void	triggerModelUpdated(ModelInfo modelInfo) trigger that a ModelInfo has been updated.
boolean	unregisterModel(java.lang.String modelName) Unregisters a model by its name.
java.util.concurrent.CompletableFuture<java.lang.String>	workerStatus() Sends model server health status to client.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

init

public static void init(ConfigManager configManager)

Initialized the global ModelManager instance.

Parameters:

configManager - the configuration

getInstance

public static ModelManager getInstance()

Returns the singleton ModelManager instance.

Returns:

the singleton ModelManager instance

registerModel

public java.util.concurrent.CompletableFuture<ModelInfo> registerModel(java.lang.String modelName,
                                                                       java.lang.String modelUrl,
                                                                       int batchSize,
                                                                       int maxBatchDelay,
                                                                       int maxIdleTime)

Registers and loads a model.

Parameters:

modelName - the name of the model for HTTP endpoint

modelUrl - the model url

batchSize - the batch size

maxBatchDelay - the maximum delay for batching

maxIdleTime - the maximum idle time of the worker threads before scaling down.

Returns:

a CompletableFuture instance

unregisterModel

public boolean unregisterModel(java.lang.String modelName)

Unregisters a model by its name.

Parameters:

modelName - the model name to be unregistered

Returns:

true if unregister success

triggerModelUpdated

public void triggerModelUpdated(ModelInfo modelInfo)

trigger that a ModelInfo has been updated. Updates model workers for this model and scales up/down all workers to match the parameters for the model.

Parameters:

modelInfo - the model that has been updated

getModels

public java.util.Map<java.lang.String,ModelInfo> getModels()

Returns the registry of all models.

Returns:

the registry of all models

getStartupModels

public java.util.Set<java.lang.String> getStartupModels()

Returns a set of models that was loaded at startup.

Returns:

a set of models that was loaded at startup

addJob

public boolean addJob(Job job)
               throws ai.djl.repository.zoo.ModelNotFoundException

Adds an inference job to the job queue. Assign the job to the next free worker.

Parameters:

job - an inference job to be executed

Returns:

true if submit success

Throws:

ai.djl.repository.zoo.ModelNotFoundException - if the model is not registered

describeModel

public DescribeModelResponse describeModel(java.lang.String modelName)
                                    throws ai.djl.repository.zoo.ModelNotFoundException

Returns a list of worker information for specified model.

Parameters:

modelName - the model to be queried

Returns:

a list of worker information for specified model

Throws:

ai.djl.repository.zoo.ModelNotFoundException - if specified model not found

workerStatus

public java.util.concurrent.CompletableFuture<java.lang.String> workerStatus()

Sends model server health status to client.

Returns:

completableFuture with eventually result in the future after async execution