Package ai.djl.training.dataset

Interface Dataset

All Known Subinterfaces:

RawDataset<T>

All Known Implementing Classes:

ArrayDataset, RandomAccessDataset

public interface Dataset

An interface to represent a set of sample data/label pairs to train a model.

See Also:

The guide to datasets, The guide to implementing a custom dataset

Nested Class Summary

Nested Classes

Modifier and Type	Interface	Description
static class	Dataset.Usage	An enum that indicates the mode - training, test or validation.

Method Summary

Modifier and Type	Method	Description
java.lang.Iterable<Batch>	getData(NDManager manager)	Fetches an iterator that can iterate through the Dataset.
default java.lang.Iterable<Batch>	getData(NDManager manager, java.util.concurrent.ExecutorService executorService)	Fetches an iterator that can iterate through the Dataset with multiple threads.
default TranslatorOptions	matchingTranslatorOptions()	Returns TranslatorOptions that match the pre-processing and post-processing of this dataset.
default void	prepare()	Prepares the dataset for use.
void	prepare(ai.djl.util.Progress progress)	Prepares the dataset for use with tracked progress.

Method Detail

getData

java.lang.Iterable<Batch> getData(NDManager manager)
                           throws java.io.IOException,
                                  TranslateException

Fetches an iterator that can iterate through the Dataset.

Parameters:

manager - the dataset to iterate through

Returns:

an Iterable of Batch that contains batches of data from the dataset

Throws:

java.io.IOException - for various exceptions depending on the dataset

TranslateException - if there is an error while processing input

getData

default java.lang.Iterable<Batch> getData(NDManager manager,
                                          java.util.concurrent.ExecutorService executorService)
                                   throws java.io.IOException,
                                          TranslateException

Fetches an iterator that can iterate through the Dataset with multiple threads.

Parameters:

manager - the dataset to iterate through

executorService - the executorService to use for multi-threading

Returns:

an Iterable of Batch that contains batches of data from the dataset

Throws:

java.io.IOException - for various exceptions depending on the dataset

TranslateException - if there is an error while processing input

prepare

default void prepare()
              throws java.io.IOException,
                     TranslateException

Prepares the dataset for use.

Throws:

java.io.IOException - for various exceptions depending on the dataset

TranslateException - if there is an error while processing input

prepare

void prepare(ai.djl.util.Progress progress)
      throws java.io.IOException,
             TranslateException

Prepares the dataset for use with tracked progress.

Parameters:

progress - the progress tracker

Throws:

java.io.IOException - for various exceptions depending on the dataset

TranslateException - if there is an error while processing input

matchingTranslatorOptions

default TranslatorOptions matchingTranslatorOptions()

Returns TranslatorOptions that match the pre-processing and post-processing of this dataset.

Returns:

matching translators or null if none defined