Package ai.djl.training.dataset

Class BulkDataIterable

java.lang.Object

ai.djl.training.dataset.DataIterable

ai.djl.training.dataset.BulkDataIterable

All Implemented Interfaces:

java.lang.Iterable<Batch>, java.util.Iterator<Batch>

public class BulkDataIterable
extends DataIterable

BulkDataIterable specializes DataIterable in using ArrayDataset.getByRange(NDManager, long, long) or ArrayDataset.getByIndices(NDManager, long...) to create Batch instances more efficiently.

Field Summary

Fields inherited from class ai.djl.training.dataset.DataIterable

dataBatchifier, dataset, device, labelBatchifier, manager, pipeline, targetPipeline

Constructor Summary

Constructors

Constructor	Description
BulkDataIterable(ArrayDataset dataset, NDManager manager, Sampler sampler, Batchifier dataBatchifier, Batchifier labelBatchifier, Pipeline pipeline, Pipeline targetPipeline, java.util.concurrent.ExecutorService executor, int preFetchNumber, Device device)	Creates a new instance of BulkDataIterable with the given parameters.

Method Summary

Modifier and Type	Method	Description
protected Batch	fetch(java.util.List<java.lang.Long> indices, int progress)
static boolean	isRange(java.util.List<java.lang.Long> indices)	Checks whether the given indices actually represents a range.

Methods inherited from class ai.djl.training.dataset.DataIterable

hasNext, iterator, next

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Methods inherited from interface java.util.Iterator

forEachRemaining, remove

Constructor Detail

BulkDataIterable

public BulkDataIterable(ArrayDataset dataset,
                        NDManager manager,
                        Sampler sampler,
                        Batchifier dataBatchifier,
                        Batchifier labelBatchifier,
                        Pipeline pipeline,
                        Pipeline targetPipeline,
                        java.util.concurrent.ExecutorService executor,
                        int preFetchNumber,
                        Device device)

Creates a new instance of BulkDataIterable with the given parameters.

Parameters:

dataset - the dataset to iterate on

manager - the manager to create the arrays

sampler - a sampler to sample data with

dataBatchifier - a batchifier for data

labelBatchifier - a batchifier for labels

pipeline - the pipeline of transforms to apply on the data

targetPipeline - the pipeline of transforms to apply on the labels

executor - an ExecutorService

preFetchNumber - the number of samples to prefetch

device - the Device

Method Detail

fetch

protected Batch fetch(java.util.List<java.lang.Long> indices,
                      int progress)
               throws java.io.IOException

Overrides:

fetch in class DataIterable

Throws:

java.io.IOException

isRange

public static boolean isRange(java.util.List<java.lang.Long> indices)

Checks whether the given indices actually represents a range.

Parameters:

indices - the indices to examine

Returns:

whether the given indices are sorted in ascending order with no gap and has at least one element