ai.djl.serving.wlm

Class ModelManager

java.lang.Object

ai.djl.serving.wlm.ModelManager

public final class ModelManager
extends java.lang.Object

A class that in charge of managing models.

Method Summary

Modifier and Type	Method and Description
boolean	addJob(Job job) Adds an inference job to the job queue.
DescribeModelResponse	describeModel(java.lang.String modelName, java.lang.String version) Returns a list of worker information for specified model.
java.util.Map<java.lang.String,Endpoint>	getEndpoints() Returns the registry of all endpoints.
static ModelManager	getInstance() Returns the singleton ModelManager instance.
ModelInfo	getModel(java.lang.String modelName, java.lang.String version, boolean predict) Returns a version of model.
java.util.Set<java.lang.String>	getStartupModels() Returns a set of models that was loaded at startup.
static void	init(ConfigManager configManager) Initialized the global ModelManager instance.
java.util.concurrent.CompletableFuture<ModelInfo>	registerModel(java.lang.String modelName, java.lang.String version, java.lang.String modelUrl, java.lang.String engineName, int gpuId, int batchSize, int maxBatchDelay, int maxIdleTime) Registers and loads a model.
ModelInfo	triggerModelUpdated(ModelInfo modelInfo) trigger that a ModelInfo has been updated.
boolean	unregisterModel(java.lang.String modelName, java.lang.String version) Unregisters a model by its name and version.
java.util.concurrent.CompletableFuture<java.lang.String>	workerStatus() Sends model server health status to client.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

init

public static void init(ConfigManager configManager)

Initialized the global ModelManager instance.

Parameters:

configManager - the configuration

getInstance

public static ModelManager getInstance()

Returns the singleton ModelManager instance.

Returns:

the singleton ModelManager instance

registerModel

public java.util.concurrent.CompletableFuture<ModelInfo> registerModel(java.lang.String modelName,
                                                                       java.lang.String version,
                                                                       java.lang.String modelUrl,
                                                                       java.lang.String engineName,
                                                                       int gpuId,
                                                                       int batchSize,
                                                                       int maxBatchDelay,
                                                                       int maxIdleTime)

Registers and loads a model.

Parameters:

modelName - the name of the model for HTTP endpoint

version - the model version

modelUrl - the model url

engineName - the engine to load the model

gpuId - the GPU device id, -1 for auto selection

batchSize - the batch size

maxBatchDelay - the maximum delay for batching

maxIdleTime - the maximum idle time of the worker threads before scaling down.

Returns:

a CompletableFuture instance

unregisterModel

public boolean unregisterModel(java.lang.String modelName,
                               java.lang.String version)

Unregisters a model by its name and version.

Parameters:

modelName - the model name to be unregistered

version - the model version

Returns:

true if unregister success

triggerModelUpdated

public ModelInfo triggerModelUpdated(ModelInfo modelInfo)

trigger that a ModelInfo has been updated. Updates model workers for this model and scales up/down all workers to match the parameters for the model.

Parameters:

modelInfo - the model that has been updated

Returns:

the model

getEndpoints

public java.util.Map<java.lang.String,Endpoint> getEndpoints()

Returns the registry of all endpoints.

Returns:

the registry of all endpoints

getModel

public ModelInfo getModel(java.lang.String modelName,
                          java.lang.String version,
                          boolean predict)

Returns a version of model.

Parameters:

modelName - the model name

version - the model version

predict - ture for selecting a model in load balance fashion

Returns:

the model

getStartupModels

public java.util.Set<java.lang.String> getStartupModels()

Returns a set of models that was loaded at startup.

Returns:

a set of models that was loaded at startup

addJob

public boolean addJob(Job job)
               throws ai.djl.repository.zoo.ModelNotFoundException

Adds an inference job to the job queue. Assign the job to the next free worker.

Parameters:

job - an inference job to be executed

Returns:

true if submit success

Throws:

ai.djl.repository.zoo.ModelNotFoundException - if the model is not registered

describeModel

public DescribeModelResponse describeModel(java.lang.String modelName,
                                           java.lang.String version)
                                    throws ai.djl.repository.zoo.ModelNotFoundException

Returns a list of worker information for specified model.

Parameters:

modelName - the model name to be queried

version - the model version to be queried

Returns:

a list of worker information for specified model

Throws:

ai.djl.repository.zoo.ModelNotFoundException - if specified model not found

workerStatus

public java.util.concurrent.CompletableFuture<java.lang.String> workerStatus()

Sends model server health status to client.

Returns:

completableFuture with eventually result in the future after async execution