org.deeplearning4j.datasets.iterator.file

Class FileDataSetIterator

java.lang.Object

org.deeplearning4j.datasets.iterator.file.BaseFileIterator<DataSet,DataSetPreProcessor>

org.deeplearning4j.datasets.iterator.file.FileDataSetIterator

All Implemented Interfaces:

Serializable, Iterator<DataSet>, DataSetIterator

public class FileDataSetIterator
extends BaseFileIterator<DataSet,DataSetPreProcessor>
implements DataSetIterator

See Also:

Serialized Form

Field Summary

Fields inherited from class org.deeplearning4j.datasets.iterator.file.BaseFileIterator

batchSize, list, order, position, preProcessor, rng

Constructor Summary

Constructors

Constructor and Description
FileDataSetIterator(File... rootDirs) Create a FileDataSetIterator with the following default settings: - Recursive: files in subdirectories are included - Randomization: order of examples is randomized with a random RNG seed - Batch size: default (as in the stored DataSets - no splitting/combining) - File extensions: no filtering - all files in directory are assumed to be a DataSet
FileDataSetIterator(File rootDir) Create a FileDataSetIterator with the following default settings: - Recursive: files in subdirectories are included - Randomization: order of examples is randomized with a random RNG seed - Batch size: default (as in the stored DataSets - no splitting/combining) - File extensions: no filtering - all files in directory are assumed to be a DataSet
FileDataSetIterator(File[] rootDirs, boolean recursive, Random rng, int batchSize, String... validExtensions) Create a FileDataSetIterator with all settings specified
FileDataSetIterator(File rootDir, boolean recursive, Random rng, int batchSize, String... validExtensions) Create a FileDataSetIterator with all settings specified
FileDataSetIterator(File rootDir, int batchSize) Create a FileDataSetIterator with the specified batch size, and the following default settings: - Recursive: files in subdirectories are included - Randomization: order of examples is randomized with a random RNG seed - File extensions: no filtering - all files in directory are assumed to be a DataSet
FileDataSetIterator(File rootDir, int batchSize, String... validExtensions) Create a FileDataSetIterator with the specified batch size, filtering based on file extensions, and the following default settings: - Recursive: files in subdirectories are included - Randomization: order of examples is randomized with a random RNG seed
FileDataSetIterator(File rootDir, String... validExtensions) Create a FileDataSetIterator with filtering based on file extensions, and the following default settings: - Recursive: files in subdirectories are included - Randomization: order of examples is randomized with a random RNG seed - Batch size: default (as in the stored DataSets - no splitting/combining)

Method Summary

Modifier and Type	Method and Description
protected void	applyPreprocessor(DataSet toPreProcess)
int	batch()
int	inputColumns()
protected DataSet	load(File f)
protected DataSet	merge(List<DataSet> toMerge)
DataSet	next(int num)
protected long	sizeOf(DataSet of)
protected List<DataSet>	split(DataSet toSplit)
int	totalOutcomes()

Methods inherited from class org.deeplearning4j.datasets.iterator.file.BaseFileIterator

asyncSupported, hasNext, mergeAndStoreRemainder, next, remove, reset, resetSupported

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.nd4j.linalg.dataset.api.iterator.DataSetIterator

asyncSupported, getLabels, getPreProcessor, reset, resetSupported, setPreProcessor

Methods inherited from interface java.util.Iterator

forEachRemaining, hasNext, next, remove

Constructor Detail

FileDataSetIterator

public FileDataSetIterator(File rootDir)

Create a FileDataSetIterator with the following default settings:
- Recursive: files in subdirectories are included
- Randomization: order of examples is randomized with a random RNG seed
- Batch size: default (as in the stored DataSets - no splitting/combining)
- File extensions: no filtering - all files in directory are assumed to be a DataSet

Parameters:

rootDir - Root directory containing the DataSet objects

FileDataSetIterator

public FileDataSetIterator(File... rootDirs)

Create a FileDataSetIterator with the following default settings:
- Recursive: files in subdirectories are included
- Randomization: order of examples is randomized with a random RNG seed
- Batch size: default (as in the stored DataSets - no splitting/combining)
- File extensions: no filtering - all files in directory are assumed to be a DataSet

Parameters:

rootDirs - Root directories containing the DataSet objects. DataSets from all of these directories will be included in the iterator output

FileDataSetIterator

public FileDataSetIterator(File rootDir,
                           int batchSize)

Create a FileDataSetIterator with the specified batch size, and the following default settings:
- Recursive: files in subdirectories are included
- Randomization: order of examples is randomized with a random RNG seed
- File extensions: no filtering - all files in directory are assumed to be a DataSet

Parameters:

rootDir - Root directory containing the saved DataSet objects

batchSize - Batch size. If > 0, DataSets will be split/recombined as required. If <= 0, DataSets will simply be loaded and returned unmodified

FileDataSetIterator

public FileDataSetIterator(File rootDir,
                           String... validExtensions)

Create a FileDataSetIterator with filtering based on file extensions, and the following default settings:
- Recursive: files in subdirectories are included
- Randomization: order of examples is randomized with a random RNG seed
- Batch size: default (as in the stored DataSets - no splitting/combining)

Parameters:

rootDir - Root directory containing the saved DataSet objects

validExtensions - May be null. If non-null, only files with one of the specified extensions will be used

FileDataSetIterator

public FileDataSetIterator(File rootDir,
                           int batchSize,
                           String... validExtensions)

Create a FileDataSetIterator with the specified batch size, filtering based on file extensions, and the following default settings:
- Recursive: files in subdirectories are included
- Randomization: order of examples is randomized with a random RNG seed

Parameters:

rootDir - Root directory containing the saved DataSet objects

batchSize - Batch size. If > 0, DataSets will be split/recombined as required. If <= 0, DataSets will simply be loaded and returned unmodified

validExtensions - May be null. If non-null, only files with one of the specified extensions will be used

FileDataSetIterator

public FileDataSetIterator(File rootDir,
                           boolean recursive,
                           Random rng,
                           int batchSize,
                           String... validExtensions)

Create a FileDataSetIterator with all settings specified

Parameters:

rootDir - Root directory containing the saved DataSet objects

recursive - If true: include files in subdirectories

rng - May be null. If non-null, use this to randomize order

batchSize - Batch size. If > 0, DataSets will be split/recombined as required. If <= 0, DataSets will simply be loaded and returned unmodified

validExtensions - May be null. If non-null, only files with one of the specified extensions will be used

FileDataSetIterator

public FileDataSetIterator(File[] rootDirs,
                           boolean recursive,
                           Random rng,
                           int batchSize,
                           String... validExtensions)

Create a FileDataSetIterator with all settings specified

Parameters:

rootDirs - Root directories containing the DataSet objects. DataSets from all of these directories will be included in the iterator output

recursive - If true: include files in subdirectories

rng - May be null. If non-null, use this to randomize order

batchSize - Batch size. If > 0, DataSets will be split/recombined as required. If <= 0, DataSets will simply be loaded and returned unmodified

validExtensions - May be null. If non-null, only files with one of the specified extensions will be used

Method Detail

load

protected DataSet load(File f)

Specified by:

load in class BaseFileIterator<DataSet,DataSetPreProcessor>

sizeOf

protected long sizeOf(DataSet of)

Specified by:

sizeOf in class BaseFileIterator<DataSet,DataSetPreProcessor>

split

protected List<DataSet> split(DataSet toSplit)

Specified by:

split in class BaseFileIterator<DataSet,DataSetPreProcessor>

merge

protected DataSet merge(List<DataSet> toMerge)

Specified by:

merge in class BaseFileIterator<DataSet,DataSetPreProcessor>

applyPreprocessor

protected void applyPreprocessor(DataSet toPreProcess)

Specified by:

applyPreprocessor in class BaseFileIterator<DataSet,DataSetPreProcessor>

next

public DataSet next(int num)

Specified by:

next in interface DataSetIterator

inputColumns

public int inputColumns()

Specified by:

inputColumns in interface DataSetIterator

totalOutcomes

public int totalOutcomes()

Specified by:

totalOutcomes in interface DataSetIterator

batch

public int batch()

Specified by:

batch in interface DataSetIterator