org.datavec.image.loader

Class CifarLoader

java.lang.Object

org.datavec.image.loader.BaseImageLoader

org.datavec.image.loader.NativeImageLoader

org.datavec.image.loader.CifarLoader

All Implemented Interfaces:

Serializable

public class CifarLoader
extends NativeImageLoader
implements Serializable

CifarLoader is loader specific for the Cifar10 dataset Reference: Learning Multiple Layers of Features from Tiny Images, Alex Krizhevsky, 2009. There is a special preProcessor used to normalize the dataset based on Sergey Zagoruyko example https://github.com/szagoruyko/cifar.torch

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static int	CHANNELS
static Map<String,String>	cifarDataMap
static boolean	DEFAULT_SHUFFLE
static boolean	DEFAULT_USE_SPECIAL_PREPROC
protected int	fileNum
protected File	fullDir
static int	HEIGHT
protected InputStream	inputStream
protected List<String>	labels
protected org.nd4j.linalg.dataset.DataSet	loadDS
protected int	loadDSIndex
protected boolean	meanStdStored
protected File	meanVarPath
static int	NUM_LABELS
static int	NUM_TEST_IMAGES
static int	NUM_TRAIN_IMAGES
protected int	numExamples
protected long	seed
protected boolean	shuffle
protected String	testFilesSerialized
protected InputStream	testInputStream
protected boolean	train
protected String	trainFilesSerialized
protected InputStream	trainInputStream
protected double	uMean
protected boolean	useSpecialPreProcessCifar
protected double	uStd
protected double	vMean
protected double	vStd
static int	WIDTH

Fields inherited from class org.datavec.image.loader.NativeImageLoader

ALLOWED_FORMATS, converter

Fields inherited from class org.datavec.image.loader.BaseImageLoader

BASE_DIR, centerCropIfNeeded, channels, height, imageTransform, log, rng, width

Constructor Summary

Constructors

Constructor and Description
CifarLoader()
CifarLoader(boolean train)
CifarLoader(boolean train, File fullPath)
CifarLoader(int height, int width, int channels, boolean train, boolean useSpecialPreProcessCifar)
CifarLoader(int height, int width, int channels, ImageTransform imgTransform, boolean train, boolean useSpecialPreProcessCifar)
CifarLoader(int height, int width, int channels, ImageTransform imgTransform, boolean train, boolean useSpecialPreProcessCifar, boolean shuffle)
CifarLoader(int height, int width, int channels, ImageTransform imgTransform, boolean train, boolean useSpecialPreProcessCifar, File fullDir, long seed, boolean shuffle)

Method Summary

Modifier and Type	Method and Description
org.nd4j.linalg.api.ndarray.INDArray	asMatrix(File f)
org.nd4j.linalg.api.ndarray.INDArray	asMatrix(InputStream inputStream)
org.nd4j.linalg.api.ndarray.INDArray	asRowVector(File f) Convert a file to a row vector
org.nd4j.linalg.api.ndarray.INDArray	asRowVector(InputStream inputStream)
org.bytedeco.javacpp.opencv_core.Mat	convertCifar(org.bytedeco.javacpp.opencv_core.Mat orgImage) Preprocess and store cifar based on successful Torch approach by Sergey Zagoruyko Reference: https://github.com/szagoruyko/cifar.torch
org.nd4j.linalg.dataset.DataSet	convertDataSet(int num)
org.nd4j.linalg.primitives.Pair<org.nd4j.linalg.api.ndarray.INDArray,org.bytedeco.javacpp.opencv_core.Mat>	convertMat(byte[] byteFeature)
protected void	generateMaps()
InputStream	getInputStream()
List<String>	getLabels()
protected void	load()
org.nd4j.linalg.dataset.DataSet	next(int batchSize)
org.nd4j.linalg.dataset.DataSet	next(int batchSize, int exampleNum)
void	normalizeCifar(File fileName) Normalize and store cifar based on successful Torch approach by Sergey Zagoruyko Reference: https://github.com/szagoruyko/cifar.torch
void	reset()
void	setInputStream()
double	varManual(org.nd4j.linalg.api.ndarray.INDArray x, double mean)

Methods inherited from class org.datavec.image.loader.NativeImageLoader

asMatrix, asMatrix, asMatrix, asMatrixView, asMatrixView, asMatrixView, asRowVector, asRowVector, asWritable, centerCropIfNeeded, fillNDArray, getAllowedFormats, scalingIfNeed, scalingIfNeed, transformImage

Methods inherited from class org.datavec.image.loader.BaseImageLoader

downloadAndUntar

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

NUM_TRAIN_IMAGES

public static final int NUM_TRAIN_IMAGES

See Also:

Constant Field Values

NUM_TEST_IMAGES

public static final int NUM_TEST_IMAGES

See Also:

Constant Field Values

NUM_LABELS

public static final int NUM_LABELS

See Also:

Constant Field Values

HEIGHT

public static final int HEIGHT

See Also:

Constant Field Values

WIDTH

public static final int WIDTH

See Also:

Constant Field Values

CHANNELS

public static final int CHANNELS

See Also:

Constant Field Values

DEFAULT_USE_SPECIAL_PREPROC

public static final boolean DEFAULT_USE_SPECIAL_PREPROC

See Also:

Constant Field Values

DEFAULT_SHUFFLE

public static final boolean DEFAULT_SHUFFLE

See Also:

Constant Field Values

fullDir

protected final File fullDir

meanVarPath

protected final File meanVarPath

trainFilesSerialized

protected final String trainFilesSerialized

testFilesSerialized

protected final String testFilesSerialized

inputStream

protected InputStream inputStream

trainInputStream

protected InputStream trainInputStream

testInputStream

protected InputStream testInputStream

labels

protected List<String> labels

cifarDataMap

public static Map<String,String> cifarDataMap

train

protected boolean train

useSpecialPreProcessCifar

protected boolean useSpecialPreProcessCifar

seed

protected long seed

shuffle

protected boolean shuffle

numExamples

protected int numExamples

uMean

protected double uMean

uStd

protected double uStd

vMean

protected double vMean

vStd

protected double vStd

meanStdStored

protected boolean meanStdStored

loadDSIndex

protected int loadDSIndex

loadDS

protected org.nd4j.linalg.dataset.DataSet loadDS

fileNum

protected int fileNum

Constructor Detail

CifarLoader

public CifarLoader()

CifarLoader

public CifarLoader(boolean train)

CifarLoader

public CifarLoader(boolean train,
                   File fullPath)

CifarLoader

public CifarLoader(int height,
                   int width,
                   int channels,
                   boolean train,
                   boolean useSpecialPreProcessCifar)

CifarLoader

public CifarLoader(int height,
                   int width,
                   int channels,
                   ImageTransform imgTransform,
                   boolean train,
                   boolean useSpecialPreProcessCifar)

CifarLoader

public CifarLoader(int height,
                   int width,
                   int channels,
                   ImageTransform imgTransform,
                   boolean train,
                   boolean useSpecialPreProcessCifar,
                   boolean shuffle)

CifarLoader

public CifarLoader(int height,
                   int width,
                   int channels,
                   ImageTransform imgTransform,
                   boolean train,
                   boolean useSpecialPreProcessCifar,
                   File fullDir,
                   long seed,
                   boolean shuffle)

Method Detail

asRowVector

public org.nd4j.linalg.api.ndarray.INDArray asRowVector(File f)
                                                 throws IOException

Description copied from class: NativeImageLoader

Convert a file to a row vector

Overrides:

asRowVector in class NativeImageLoader

Parameters:

f - the image to convert

Returns:

the flattened image

Throws:

IOException

asRowVector

public org.nd4j.linalg.api.ndarray.INDArray asRowVector(InputStream inputStream)
                                                 throws IOException

Overrides:

asRowVector in class NativeImageLoader

Throws:

IOException

asMatrix

public org.nd4j.linalg.api.ndarray.INDArray asMatrix(File f)
                                              throws IOException

Overrides:

asMatrix in class NativeImageLoader

Throws:

IOException

asMatrix

public org.nd4j.linalg.api.ndarray.INDArray asMatrix(InputStream inputStream)
                                              throws IOException

Overrides:

asMatrix in class NativeImageLoader

Throws:

IOException

generateMaps

protected void generateMaps()

load

protected void load()

convertCifar

public org.bytedeco.javacpp.opencv_core.Mat convertCifar(org.bytedeco.javacpp.opencv_core.Mat orgImage)

Preprocess and store cifar based on successful Torch approach by Sergey Zagoruyko Reference: https://github.com/szagoruyko/cifar.torch

normalizeCifar

public void normalizeCifar(File fileName)

Normalize and store cifar based on successful Torch approach by Sergey Zagoruyko Reference: https://github.com/szagoruyko/cifar.torch

convertMat

public org.nd4j.linalg.primitives.Pair<org.nd4j.linalg.api.ndarray.INDArray,org.bytedeco.javacpp.opencv_core.Mat> convertMat(byte[] byteFeature)

convertDataSet

public org.nd4j.linalg.dataset.DataSet convertDataSet(int num)

varManual

public double varManual(org.nd4j.linalg.api.ndarray.INDArray x,
                        double mean)

next

public org.nd4j.linalg.dataset.DataSet next(int batchSize)

next

public org.nd4j.linalg.dataset.DataSet next(int batchSize,
                                            int exampleNum)

getInputStream

public InputStream getInputStream()

setInputStream

public void setInputStream()

getLabels

public List<String> getLabels()

reset

public void reset()