org.deeplearning4j.datasets.iterator.file

Class FileMultiDataSetIterator

java.lang.Object

org.deeplearning4j.datasets.iterator.file.BaseFileIterator<MultiDataSet,MultiDataSetPreProcessor>

org.deeplearning4j.datasets.iterator.file.FileMultiDataSetIterator

All Implemented Interfaces:

Serializable, Iterator<MultiDataSet>, MultiDataSetIterator

public class FileMultiDataSetIterator
extends BaseFileIterator<MultiDataSet,MultiDataSetPreProcessor>
implements MultiDataSetIterator

See Also:

Serialized Form

Field Summary

Fields inherited from class org.deeplearning4j.datasets.iterator.file.BaseFileIterator

batchSize, list, order, position, preProcessor, rng

Constructor Summary

Constructors

Constructor and Description
FileMultiDataSetIterator(File... rootDirs) Create a FileMultiDataSetIterator with the following default settings: - Recursive: files in subdirectories are included - Randomization: order of examples is randomized with a random RNG seed - Batch size: default (as in the stored DataSets - no splitting/combining) - File extensions: no filtering - all files in directory are assumed to be a DataSet
FileMultiDataSetIterator(File rootDir) Create a FileMultiDataSetIterator with the following default settings: - Recursive: files in subdirectories are included - Randomization: order of examples is randomized with a random RNG seed - Batch size: default (as in the stored DataSets - no splitting/combining) - File extensions: no filtering - all files in directory are assumed to be a DataSet
FileMultiDataSetIterator(File[] rootDirs, boolean recursive, Random rng, int batchSize, String... validExtensions) Create a FileMultiDataSetIterator with all settings specified
FileMultiDataSetIterator(File rootDir, boolean recursive, Random rng, int batchSize, String... validExtensions) Create a FileMultiDataSetIterator with all settings specified
FileMultiDataSetIterator(File rootDir, int batchSize) Create a FileMultiDataSetIterator with the specified batch size, and the following default settings: - Recursive: files in subdirectories are included - Randomization: order of examples is randomized with a random RNG seed - File extensions: no filtering - all files in directory are assumed to be a DataSet
FileMultiDataSetIterator(File rootDir, int batchSize, String... validExtensions) Create a FileMultiDataSetIterator with the specified batch size, filtering based on file extensions, and the following default settings: - Recursive: files in subdirectories are included - Randomization: order of examples is randomized with a random RNG seed
FileMultiDataSetIterator(File rootDir, String... validExtensions) Create a FileMultiDataSetIterator with filtering based on file extensions, and the following default settings: - Recursive: files in subdirectories are included - Randomization: order of examples is randomized with a random RNG seed - Batch size: default (as in the stored DataSets - no splitting/combining)

Method Summary

Modifier and Type	Method and Description
protected void	applyPreprocessor(MultiDataSet toPreProcess)
protected MultiDataSet	load(File f)
MultiDataSet	merge(List<MultiDataSet> toMerge)
MultiDataSet	next(int num)
protected long	sizeOf(MultiDataSet of)
protected List<MultiDataSet>	split(MultiDataSet toSplit)

Methods inherited from class org.deeplearning4j.datasets.iterator.file.BaseFileIterator

asyncSupported, hasNext, mergeAndStoreRemainder, next, remove, reset, resetSupported

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.nd4j.linalg.dataset.api.iterator.MultiDataSetIterator

asyncSupported, getPreProcessor, reset, resetSupported, setPreProcessor

Methods inherited from interface java.util.Iterator

forEachRemaining, hasNext, next, remove

Constructor Detail

FileMultiDataSetIterator

public FileMultiDataSetIterator(File rootDir)

Create a FileMultiDataSetIterator with the following default settings:
- Recursive: files in subdirectories are included
- Randomization: order of examples is randomized with a random RNG seed
- Batch size: default (as in the stored DataSets - no splitting/combining)
- File extensions: no filtering - all files in directory are assumed to be a DataSet

Parameters:

rootDir - Root directory containing the DataSet objects

FileMultiDataSetIterator

public FileMultiDataSetIterator(File... rootDirs)

Create a FileMultiDataSetIterator with the following default settings:
- Recursive: files in subdirectories are included
- Randomization: order of examples is randomized with a random RNG seed
- Batch size: default (as in the stored DataSets - no splitting/combining)
- File extensions: no filtering - all files in directory are assumed to be a DataSet

Parameters:

rootDirs - Root directories containing the MultiDataSet objects. MultiDataSets from all of these directories will be included in the iterator output

FileMultiDataSetIterator

public FileMultiDataSetIterator(File rootDir,
                                int batchSize)

Create a FileMultiDataSetIterator with the specified batch size, and the following default settings:
- Recursive: files in subdirectories are included
- Randomization: order of examples is randomized with a random RNG seed
- File extensions: no filtering - all files in directory are assumed to be a DataSet

Parameters:

rootDir - Root directory containing the saved DataSet objects

batchSize - Batch size. If > 0, DataSets will be split/recombined as required. If <= 0, DataSets will simply be loaded and returned unmodified

FileMultiDataSetIterator

public FileMultiDataSetIterator(File rootDir,
                                String... validExtensions)

Create a FileMultiDataSetIterator with filtering based on file extensions, and the following default settings:
- Recursive: files in subdirectories are included
- Randomization: order of examples is randomized with a random RNG seed
- Batch size: default (as in the stored DataSets - no splitting/combining)

Parameters:

rootDir - Root directory containing the saved DataSet objects

validExtensions - May be null. If non-null, only files with one of the specified extensions will be used

FileMultiDataSetIterator

public FileMultiDataSetIterator(File rootDir,
                                int batchSize,
                                String... validExtensions)

Create a FileMultiDataSetIterator with the specified batch size, filtering based on file extensions, and the following default settings:
- Recursive: files in subdirectories are included
- Randomization: order of examples is randomized with a random RNG seed

Parameters:

rootDir - Root directory containing the saved DataSet objects

batchSize - Batch size. If > 0, DataSets will be split/recombined as required. If <= 0, DataSets will simply be loaded and returned unmodified

validExtensions - May be null. If non-null, only files with one of the specified extensions will be used

FileMultiDataSetIterator

public FileMultiDataSetIterator(File rootDir,
                                boolean recursive,
                                Random rng,
                                int batchSize,
                                String... validExtensions)

Create a FileMultiDataSetIterator with all settings specified

Parameters:

rootDir - Root directory containing the saved DataSet objects

recursive - If true: include files in subdirectories

rng - May be null. If non-null, use this to randomize order

batchSize - Batch size. If > 0, DataSets will be split/recombined as required. If <= 0, DataSets will simply be loaded and returned unmodified

validExtensions - May be null. If non-null, only files with one of the specified extensions will be used

FileMultiDataSetIterator

public FileMultiDataSetIterator(File[] rootDirs,
                                boolean recursive,
                                Random rng,
                                int batchSize,
                                String... validExtensions)

Create a FileMultiDataSetIterator with all settings specified

Parameters:

rootDirs - Root directories containing the MultiDataSet objects. MultiDataSets from all of these directories will be included in the iterator output

recursive - If true: include files in subdirectories

rng - May be null. If non-null, use this to randomize order

batchSize - Batch size. If > 0, DataSets will be split/recombined as required. If <= 0, DataSets will simply be loaded and returned unmodified

validExtensions - May be null. If non-null, only files with one of the specified extensions will be used

Method Detail

load

protected MultiDataSet load(File f)

Specified by:

load in class BaseFileIterator<MultiDataSet,MultiDataSetPreProcessor>

sizeOf

protected long sizeOf(MultiDataSet of)

Specified by:

sizeOf in class BaseFileIterator<MultiDataSet,MultiDataSetPreProcessor>

split

protected List<MultiDataSet> split(MultiDataSet toSplit)

Specified by:

split in class BaseFileIterator<MultiDataSet,MultiDataSetPreProcessor>

merge

public MultiDataSet merge(List<MultiDataSet> toMerge)

Specified by:

merge in class BaseFileIterator<MultiDataSet,MultiDataSetPreProcessor>

applyPreprocessor

protected void applyPreprocessor(MultiDataSet toPreProcess)

Specified by:

applyPreprocessor in class BaseFileIterator<MultiDataSet,MultiDataSetPreProcessor>

next

public MultiDataSet next(int num)

Specified by:

next in interface MultiDataSetIterator