Package ai.djl.inference

Class Predictor<I,O>

java.lang.Object

ai.djl.inference.Predictor<I,O>

Type Parameters:

I - the input type

O - the output type

All Implemented Interfaces:

java.lang.AutoCloseable

public class Predictor<I,O>
extends java.lang.Object
implements java.lang.AutoCloseable

The Predictor interface provides a session for model inference.

You can use a Predictor, with a specified Translator, to perform inference on a Model. The following is example code that uses Predictor:

 Model model = Model.load(modelDir, modelName);

 // User must implement Translator interface, read Translator for detail.
 Translator<String, String> translator = new MyTranslator();

 try (Predictor<String, String> predictor = model.newPredictor(translator)) {
   String result = predictor.predict("What's up");
 }
 

See the tutorials on:

• Inference with a custom trained model
• Inference with a model zoo model
• Inference with an MXNet model

See Also:

Model, Translator, The guide on memory management, The guide on running multi-threaded inference, The guide on inference performance optimization

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
protected class	Predictor.PredictorContext

Field Summary

Fields

Modifier and Type	Field	Description
protected Block	block
protected NDManager	manager
protected Metrics	metrics
protected Model	model
protected ParameterStore	parameterStore
protected boolean	prepared
protected long	timestamp
protected Translator<I,O>	translator

Constructor Summary

Constructors

Constructor	Description
Predictor(Model model, Translator<I,O> translator, Device device, boolean copy)	Creates a new instance of BasePredictor with the given Model and Translator.

Method Summary

Modifier and Type	Method	Description
java.util.List<O>	batchPredict(java.util.List<I> inputs)	Predicts a batch for inference.
void	close()
protected void	finalize()
O	predict(I input)	Predicts an item for inference.
protected NDList	predictInternal(TranslatorContext ctx, NDList ndList)	Predicts an item for inference.
void	setMetrics(Metrics metrics)	Attaches a Metrics param to use for benchmark.
StreamingTranslator.StreamOutput<O>	streamingPredict(I input)	Predicts an item for inference.
boolean	supportsStreaming()	Returns true if streaming is supported by the predictor, block, and translator.

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

translator

protected Translator<I,O> translator

timestamp

protected long timestamp

prepared

protected boolean prepared

model

protected Model model

manager

protected NDManager manager

metrics

protected Metrics metrics

block

protected Block block

parameterStore

protected ParameterStore parameterStore

Constructor Detail

Predictor

public Predictor(Model model,
                 Translator<I,O> translator,
                 Device device,
                 boolean copy)

Creates a new instance of BasePredictor with the given Model and Translator.

Parameters:

model - the model on which the predictions are based

translator - the translator to be used

device - the device for prediction

copy - whether to copy the parameters to the parameter store. If the device changes, it will copy regardless

Method Detail

predict

public O predict(I input)
          throws TranslateException

Predicts an item for inference.

Parameters:

input - the input

Returns:

the output object defined by the user

Throws:

TranslateException - if an error occurs during prediction

predictInternal

protected NDList predictInternal(TranslatorContext ctx,
                                 NDList ndList)
                          throws TranslateException

Predicts an item for inference.

Parameters:

ctx - the context for the Predictor.

ndList - the input NDList

Returns:

the output NDList

Throws:

TranslateException - if an error occurs during prediction

batchPredict

public java.util.List<O> batchPredict(java.util.List<I> inputs)
                               throws TranslateException

Predicts a batch for inference.

Parameters:

inputs - a list of inputs

Returns:

a list of output objects defined by the user

Throws:

TranslateException - if an error occurs during prediction

streamingPredict

public StreamingTranslator.StreamOutput<O> streamingPredict(I input)
                                                     throws TranslateException

Predicts an item for inference.

Parameters:

input - the input

Returns:

the output object defined by the user

Throws:

TranslateException - if an error occurs during prediction

supportsStreaming

public boolean supportsStreaming()

Returns true if streaming is supported by the predictor, block, and translator.

Returns:

true if streaming is supported by the predictor, block, and translator

setMetrics

public void setMetrics(Metrics metrics)

Attaches a Metrics param to use for benchmark.

Parameters:

metrics - the Metrics class

close

public void close()

Specified by:

close in interface java.lang.AutoCloseable

finalize

protected void finalize()
                 throws java.lang.Throwable

Overrides:

finalize in class java.lang.Object

Throws:

java.lang.Throwable