Package ai.djl.inference.streaming
Interface StreamingTranslator<I,O>
-
- Type Parameters:
I- the input typeO- the output type
- All Superinterfaces:
PostProcessor<O>,PreProcessor<I>,Translator<I,O>
- All Known Subinterfaces:
ServingTranslator
public interface StreamingTranslator<I,O> extends Translator<I,O>
An expansion of theTranslatorwith postProcessing for theStreamingBlock(used byPredictor.streamingPredict(Object).
-
-
Nested Class Summary
Nested Classes Modifier and Type Interface Description static classStreamingTranslator.StreamOutput<O>AStreamingTranslator.StreamOutputrepresents a streamable output type (either iterative or asynchronous).static classStreamingTranslator.SupportWhat types ofStreamingTranslator.StreamOutputs are supported by aStreamingTranslator.
-
Method Summary
All Methods Instance Methods Abstract Methods Default Methods Modifier and Type Method Description StreamingTranslator.SupportgetSupport()Returns what kind ofStreamingTranslator.StreamOutputthisStreamingTranslatorsupports.StreamingTranslator.StreamOutput<O>processStreamOutput(TranslatorContext ctx, java.util.stream.Stream<NDList> list)Processes the output NDList to the corresponding output object.default booleansupportsAsync()Returns whether theStreamingTranslatorsupports iterative output.default booleansupportsIterative()Returns whether theStreamingTranslatorsupports iterative output.-
Methods inherited from interface ai.djl.translate.PostProcessor
processOutput
-
Methods inherited from interface ai.djl.translate.PreProcessor
processInput
-
Methods inherited from interface ai.djl.translate.Translator
getBatchifier, getExpansions, prepare, toBatchTranslator, toBatchTranslator
-
-
-
-
Method Detail
-
processStreamOutput
StreamingTranslator.StreamOutput<O> processStreamOutput(TranslatorContext ctx, java.util.stream.Stream<NDList> list) throws java.lang.Exception
Processes the output NDList to the corresponding output object.- Parameters:
ctx- the toolkit used for post-processinglist- the output NDList after inference, usually immutable in engines like PyTorch. @see Issue 1774- Returns:
- the output object of expected type
- Throws:
java.lang.Exception- if an error occurs during processing output
-
getSupport
StreamingTranslator.Support getSupport()
Returns what kind ofStreamingTranslator.StreamOutputthisStreamingTranslatorsupports.- Returns:
- what kind of
StreamingTranslator.StreamOutputthisStreamingTranslatorsupports
-
supportsIterative
default boolean supportsIterative()
Returns whether theStreamingTranslatorsupports iterative output.- Returns:
- whether the
StreamingTranslatorsupports iterative output - See Also:
StreamingTranslator.StreamOutput.getIterativeOutput()
-
supportsAsync
default boolean supportsAsync()
Returns whether theStreamingTranslatorsupports iterative output.- Returns:
- whether the
StreamingTranslatorsupports iterative output - See Also:
StreamingTranslator.StreamOutput.getAsyncOutput()
-
-