ai.djl.training.dataset

Class RandomAccessDataset

java.lang.Object

ai.djl.training.dataset.RandomAccessDataset

All Implemented Interfaces:

Dataset, java.util.RandomAccess

Direct Known Subclasses:

ArrayDataset

public abstract class RandomAccessDataset
extends java.lang.Object
implements Dataset, java.util.RandomAccess

RandomAccessDataset represent the dataset that support random access reads. i.e. it could access a specific data item given the index.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	RandomAccessDataset.BaseBuilder<T extends RandomAccessDataset.BaseBuilder> The Builder to construct a RandomAccessDataset.

Nested classes/interfaces inherited from interface ai.djl.training.dataset.Dataset

Dataset.Usage

Field Summary

Fields

Modifier and Type	Field and Description
protected Batchifier	batchifier
protected Device	device
protected java.util.concurrent.ExecutorService	executor
protected long	maxIteration
protected Pipeline	pipeline
protected int	prefetchNumber
protected Sampler	sampler
protected Pipeline	targetPipeline

Constructor Summary

Constructors

Constructor and Description
RandomAccessDataset(RandomAccessDataset.BaseBuilder<?> builder) Creates a new instance of RandomAccessDataset with the given necessary configurations.

Method Summary

Modifier and Type	Method and Description
abstract Record	get(NDManager manager, long index) Gets the Record for the given index from the dataset.
java.lang.Iterable<Batch>	getData(NDManager manager) Fetches an iterator that can iterate through the Dataset.
long	getNumIterations() Returns the number of iteration of the batch iterable.
RandomAccessDataset[]	randomSplit(int... ratio) Splits the dataset set into multiple portions.
abstract long	size() Returns the size of this Dataset.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

sampler

protected Sampler sampler

batchifier

protected Batchifier batchifier

pipeline

protected Pipeline pipeline

targetPipeline

protected Pipeline targetPipeline

executor

protected java.util.concurrent.ExecutorService executor

prefetchNumber

protected int prefetchNumber

maxIteration

protected long maxIteration

device

protected Device device

Constructor Detail

RandomAccessDataset

public RandomAccessDataset(RandomAccessDataset.BaseBuilder<?> builder)

Creates a new instance of RandomAccessDataset with the given necessary configurations.

Parameters:

builder - a builder with the necessary configurations

Method Detail

get

public abstract Record get(NDManager manager,
                           long index)
                    throws java.io.IOException

Gets the Record for the given index from the dataset.

Parameters:

manager - the manager used to create the arrays

index - the index of the requested data item

Returns:

a Record that contains the data and label of the requested data item

Throws:

java.io.IOException - if an I/O error occurs

getData

public java.lang.Iterable<Batch> getData(NDManager manager)

Fetches an iterator that can iterate through the Dataset.

Specified by:

getData in interface Dataset

Parameters:

manager - the dataset to iterate through

Returns:

an Iterable of Batch that contains batches of data from the dataset

size

public abstract long size()

Returns the size of this Dataset.

Returns:

the size of this Dataset

getNumIterations

public long getNumIterations()

Returns the number of iteration of the batch iterable.

Returns:

the number of iteration of the batch iterable, -1 if number of iterations is unknown

randomSplit

public RandomAccessDataset[] randomSplit(int... ratio)

Splits the dataset set into multiple portions.

Parameters:

ratio - the ratio of each sub dataset

Returns:

an array of the sub dataset