org.nd4j.linalg.dataset

Class DataSet

java.lang.Object

org.nd4j.linalg.dataset.DataSet

All Implemented Interfaces:

Serializable, Iterable<DataSet>, DataSet

public class DataSet
extends Object
implements DataSet

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
DataSet()
DataSet(INDArray first, INDArray second) Creates a dataset with the specified input matrix and labels
DataSet(INDArray features, INDArray labels, INDArray featuresMask, INDArray labelsMask) Create a dataset with the specified input INDArray and labels (output) INDArray, plus (optionally) mask arrays for the features and labels

Method Summary

Modifier and Type	Method and Description
void	addFeatureVector(INDArray toAdd) Adds a feature for each example on to the current feature vector
void	addFeatureVector(INDArray feature, int example) The feature to add, and the example/row number
void	addRow(DataSet d, int i)
List<DataSet>	asList() Extract each example in the DataSet into its own DataSet object, and return all of them as a list
List<DataSet>	batchBy(int num) Partitions a dataset in to mini batches where each dataset in each list is of the specified number of examples
List<DataSet>	batchByNumLabels()
void	binarize() Same as calling binarize(0)
void	binarize(double cutoff) Binarizes the dataset such that any number greater than cutoff is 1 otherwise zero
DataSet	copy() Clone the dataset
List<DataSet>	dataSetBatches(int num) Partitions the data transform by the specified number.
void	detach() This method detaches this DataSet from current Workspace (if any)
void	divideBy(int num) Divide the features by a scalar
static DataSet	empty() Returns a single dataset (all fields are null)
boolean	equals(Object o)
INDArray	exampleMaxs()
INDArray	exampleMeans()
INDArray	exampleSums()
void	filterAndStrip(int[] labels) Strips the dataset down to the specified labels and remaps them
DataSet	filterBy(int[] labels) Strips the data transform of all but the passed in labels
DataSet	get(int i) Gets a copy of example i
DataSet	get(int[] i) Gets a copy of example i
List<String>	getColumnNames() Deprecated.
List<Serializable>	getExampleMetaData() Get the example metadata, or null if no metadata has been set
<T extends Serializable> List<T>	getExampleMetaData(Class<T> metaDataType) Get the example metadata, or null if no metadata has been set Note: this method results in an unchecked cast - care should be taken when using this!
INDArray	getFeatures() Returns the features array for the DataSet
INDArray	getFeaturesMaskArray() Input mask array: a mask array for input, where each value is in {0,1} in order to specify whether an input is actually present or not.
String	getLabelName(int idx)
List<String>	getLabelNames() Deprecated.
List<String>	getLabelNames(INDArray idxs)
List<String>	getLabelNamesList() Gets the optional label names
INDArray	getLabels() Returns the labels for the dataset
INDArray	getLabelsMaskArray() Labels (output) mask array: a mask array for input, where each value is in {0,1} in order to specify whether an output is actually present or not.
long	getMemoryFootprint() This method returns memory used by this DataSet
DataSet	getRange(int from, int to)
int	hashCode()
boolean	hasMaskArrays() Whether the labels or input (features) mask arrays are present for this DataSet
String	id()
boolean	isEmpty()
boolean	isPreProcessed()
DataSetIterator	iterateWithMiniBatches()
Iterator<DataSet>	iterator()
Map<Integer,Double>	labelCounts() Calculate and return a count of each label, by index.
void	load(File from) Load the contents of the DataSet from the specified File.
void	load(InputStream from) Load the contents of the DataSet from the specified InputStream.
void	markAsPreProcessed()
static DataSet	merge(List<? extends DataSet> data) Merge the list of datasets in to one list.
void	migrate() This method migrates this DataSet into current Workspace (if any)
void	multiplyBy(double num) Multiply the features by a scalar
void	normalize() Normalize this DataSet to mean 0, stdev 1 per input.
void	normalizeZeroMeanZeroUnitVariance() Deprecated.
int	numExamples() Number of examples in the DataSet
int	numInputs() The number of inputs in the feature matrix
int	numOutcomes() Returns the number of outcomes (size of the labels array for each example)
int	outcome()
DataSet	reshape(int rows, int cols) Reshapes the input in to the given rows and columns
void	roundToTheNearest(int roundTo)
DataSet	sample(int numSamples) Sample without replacement and a random rng
DataSet	sample(int numSamples, boolean withReplacement) Sample a dataset numSamples times
DataSet	sample(int numSamples, Random rng) Sample without replacement
DataSet	sample(int numSamples, Random rng, boolean withReplacement) Sample a dataset
void	save(File to) Save this DataSet to a file.
void	save(OutputStream to) Write the contents of this DataSet to the specified OutputStream
void	scale() Divides the input data transform by the max number in each row
void	scaleMinAndMax(double min, double max)
void	setColumnNames(List<String> columnNames) Deprecated.
void	setExampleMetaData(List<? extends Serializable> exampleMetaData) Set the metadata for this DataSet By convention: the metadata can be any serializable object, one per example in the DataSet
void	setFeatures(INDArray features) Set the features array for the DataSet
void	setFeaturesMaskArray(INDArray featuresMask) Set the features mask array in this DataSet
void	setLabelNames(List<String> labelNames) Sets the label names, will throw an exception if the passed in label names doesn't equal the number of outcomes
void	setLabels(INDArray labels)
void	setLabelsMaskArray(INDArray labelsMask) Set the labels mask array in this data set
void	setNewNumberOfLabels(int labels) Clears the outcome matrix setting a new number of labels
void	setOutcome(int example, int label) Sets the outcome of a particular example
void	shuffle() Shuffle the order of the rows in the DataSet.
void	shuffle(long seed) Shuffles the dataset in place, given a seed for a random number generator.
List<DataSet>	sortAndBatchByNumLabels() Sorts the dataset by label: Splits the data transform such that examples are sorted by their labels.
void	sortByLabel() Organizes the dataset to minimize sampling error while still allowing efficient batching.
SplitTestAndTrain	splitTestAndTrain(double fractionTrain) SplitV the DataSet into two DataSets randomly
SplitTestAndTrain	splitTestAndTrain(int numHoldout) Splits a dataset in to test and train
SplitTestAndTrain	splitTestAndTrain(int numHoldout, Random rng) Splits a dataset in to test and train randomly.
void	squishToRange(double min, double max) Squeezes input data to a max and a min
MultiDataSet	toMultiDataSet()
String	toString()
void	validate()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Constructor Detail

DataSet

public DataSet()

DataSet

public DataSet(INDArray first,
               INDArray second)

Creates a dataset with the specified input matrix and labels

Parameters:

first - the feature matrix

second - the labels (these should be binarized label matrices such that the specified label has a value of 1 in the desired column with the label)

DataSet

public DataSet(INDArray features,
               INDArray labels,
               INDArray featuresMask,
               INDArray labelsMask)

Create a dataset with the specified input INDArray and labels (output) INDArray, plus (optionally) mask arrays for the features and labels

Parameters:

features - Features (input)

labels - Labels (output)

featuresMask - Mask array for features, may be null

labelsMask - Mask array for labels, may be null

Method Detail

getExampleMetaData

public List<Serializable> getExampleMetaData()

Description copied from interface: DataSet

Get the example metadata, or null if no metadata has been set

Specified by:

getExampleMetaData in interface DataSet

Returns:

List of metadata instances

See Also:

#getExampleMetaData(Class)} for convenience method for types

getExampleMetaData

public <T extends Serializable> List<T> getExampleMetaData(Class<T> metaDataType)

Description copied from interface: DataSet

Get the example metadata, or null if no metadata has been set
Note: this method results in an unchecked cast - care should be taken when using this!

Specified by:

getExampleMetaData in interface DataSet

Type Parameters:

T - Type of metadata

Parameters:

metaDataType - Class of the metadata (used for opType information)

Returns:

List of metadata objects

setExampleMetaData

public void setExampleMetaData(List<? extends Serializable> exampleMetaData)

Description copied from interface: DataSet

Set the metadata for this DataSet
By convention: the metadata can be any serializable object, one per example in the DataSet

Specified by:

setExampleMetaData in interface DataSet

Parameters:

exampleMetaData - Example metadata to set

isPreProcessed

public boolean isPreProcessed()

markAsPreProcessed

public void markAsPreProcessed()

empty

public static DataSet empty()

Returns a single dataset (all fields are null)

Returns:

an empty dataset (all fields are null)

merge

public static DataSet merge(List<? extends DataSet> data)

Merge the list of datasets in to one list. All the rows are merged in to one dataset

Parameters:

data - the data to merge

Returns:

a single dataset

getRange

public DataSet getRange(int from,
                        int to)

Specified by:

getRange in interface DataSet

load

public void load(InputStream from)

Description copied from interface: DataSet

Load the contents of the DataSet from the specified InputStream. The current contents of the DataSet (if any) will be replaced.
The InputStream should contain a DataSet that has been serialized with DataSet.save(OutputStream)

Specified by:

load in interface DataSet

Parameters:

from - InputStream to load the DataSet from

load

public void load(File from)

Description copied from interface: DataSet

Load the contents of the DataSet from the specified File. The current contents of the DataSet (if any) will be replaced.
The InputStream should contain a DataSet that has been serialized with DataSet.save(File)

Specified by:

load in interface DataSet

Parameters:

from - File to load the DataSet from

save

public void save(OutputStream to)

Description copied from interface: DataSet

Write the contents of this DataSet to the specified OutputStream

Specified by:

save in interface DataSet

Parameters:

to - OutputStream to save the DataSet to

save

public void save(File to)

Description copied from interface: DataSet

Save this DataSet to a file. Can be loaded again using

Specified by:

save in interface DataSet

Parameters:

to - File to sa

iterateWithMiniBatches

public DataSetIterator iterateWithMiniBatches()

Specified by:

iterateWithMiniBatches in interface DataSet

id

public String id()

Specified by:

id in interface DataSet

getFeatures

public INDArray getFeatures()

Description copied from interface: DataSet

Returns the features array for the DataSet

Specified by:

getFeatures in interface DataSet

Returns:

features array

setFeatures

public void setFeatures(INDArray features)

Description copied from interface: DataSet

Set the features array for the DataSet

Specified by:

setFeatures in interface DataSet

Parameters:

features - Features to set

labelCounts

public Map<Integer,Double> labelCounts()

Description copied from interface: DataSet

Calculate and return a count of each label, by index. Assumes labels are a one-hot INDArray, for classification

Specified by:

labelCounts in interface DataSet

Returns:

Map of countsn

copy

public DataSet copy()

Clone the dataset

Specified by:

copy in interface DataSet

Returns:

a clone of the dataset

reshape

public DataSet reshape(int rows,
                       int cols)

Reshapes the input in to the given rows and columns

Specified by:

reshape in interface DataSet

Parameters:

rows - the row size

cols - the column size

Returns:

a copy of this data op with the input resized

multiplyBy

public void multiplyBy(double num)

Description copied from interface: DataSet

Multiply the features by a scalar

Specified by:

multiplyBy in interface DataSet

divideBy

public void divideBy(int num)

Description copied from interface: DataSet

Divide the features by a scalar

Specified by:

divideBy in interface DataSet

shuffle

public void shuffle()

Description copied from interface: DataSet

Shuffle the order of the rows in the DataSet. Note that this generally won't make any difference in practice unless the DataSet is later split.

Specified by:

shuffle in interface DataSet

shuffle

public void shuffle(long seed)

Shuffles the dataset in place, given a seed for a random number generator. For reproducibility This will modify the dataset in place!!

Parameters:

seed - Seed to use for the random Number Generator

squishToRange

public void squishToRange(double min,
                          double max)

Squeezes input data to a max and a min

Specified by:

squishToRange in interface DataSet

Parameters:

min - the min value to occur in the dataset

max - the max value to ccur in the dataset

scaleMinAndMax

public void scaleMinAndMax(double min,
                           double max)

Specified by:

scaleMinAndMax in interface DataSet

scale

public void scale()

Divides the input data transform by the max number in each row

Specified by:

scale in interface DataSet

addFeatureVector

public void addFeatureVector(INDArray toAdd)

Adds a feature for each example on to the current feature vector

Specified by:

addFeatureVector in interface DataSet

Parameters:

toAdd - the feature vector to add

addFeatureVector

public void addFeatureVector(INDArray feature,
                             int example)

The feature to add, and the example/row number

Specified by:

addFeatureVector in interface DataSet

Parameters:

feature - the feature vector to add

example - the number of the example to append to

normalize

public void normalize()

Description copied from interface: DataSet

Normalize this DataSet to mean 0, stdev 1 per input. This calculates statistics based on the values in a single DataSet only. For normalization over multiple DataSet objects, use NormalizerStandardize

Specified by:

normalize in interface DataSet

binarize

public void binarize()

Same as calling binarize(0)

Specified by:

binarize in interface DataSet

binarize

public void binarize(double cutoff)

Binarizes the dataset such that any number greater than cutoff is 1 otherwise zero

Specified by:

binarize in interface DataSet

Parameters:

cutoff - the cutoff point

normalizeZeroMeanZeroUnitVariance

@Deprecated
public void normalizeZeroMeanZeroUnitVariance()

Deprecated.

Specified by:

normalizeZeroMeanZeroUnitVariance in interface DataSet

numInputs

public int numInputs()

The number of inputs in the feature matrix

Specified by:

numInputs in interface DataSet

Returns:

validate

public void validate()

Specified by:

validate in interface DataSet

outcome

public int outcome()

Specified by:

outcome in interface DataSet

setNewNumberOfLabels

public void setNewNumberOfLabels(int labels)

Clears the outcome matrix setting a new number of labels

Specified by:

setNewNumberOfLabels in interface DataSet

Parameters:

labels - the number of labels/columns in the outcome matrix Note that this clears the labels for each example

setOutcome

public void setOutcome(int example,
                       int label)

Sets the outcome of a particular example

Specified by:

setOutcome in interface DataSet

Parameters:

example - the example to transform

label - the label of the outcome

get

public DataSet get(int i)

Gets a copy of example i

Specified by:

get in interface DataSet

Parameters:

i - the example to getFromOrigin

Returns:

the example at i (one example)

get

public DataSet get(int[] i)

Gets a copy of example i

Specified by:

get in interface DataSet

Parameters:

i - the example to getFromOrigin

Returns:

the example at i (one example)

batchBy

public List<DataSet> batchBy(int num)

Partitions a dataset in to mini batches where each dataset in each list is of the specified number of examples

Specified by:

batchBy in interface DataSet

Parameters:

num - the number to split by

Returns:

the partitioned datasets

filterBy

public DataSet filterBy(int[] labels)

Strips the data transform of all but the passed in labels

Specified by:

filterBy in interface DataSet

Parameters:

labels - strips the data transform of all but the passed in labels

Returns:

the dataset with only the specified labels

filterAndStrip

public void filterAndStrip(int[] labels)

Strips the dataset down to the specified labels and remaps them

Specified by:

filterAndStrip in interface DataSet

Parameters:

labels - the labels to strip down to

dataSetBatches

public List<DataSet> dataSetBatches(int num)

Partitions the data transform by the specified number.

Specified by:

dataSetBatches in interface DataSet

Parameters:

num - the number to split by

Returns:

the partitioned data transform

sortAndBatchByNumLabels

public List<DataSet> sortAndBatchByNumLabels()

Sorts the dataset by label: Splits the data transform such that examples are sorted by their labels. A ten label dataset would produce lists with batches like the following: x1 y = 1 x2 y = 2 ... x10 y = 10

Specified by:

sortAndBatchByNumLabels in interface DataSet

Returns:

a list of data sets partitioned by outcomes

batchByNumLabels

public List<DataSet> batchByNumLabels()

Specified by:

batchByNumLabels in interface DataSet

asList

public List<DataSet> asList()

Description copied from interface: DataSet

Extract each example in the DataSet into its own DataSet object, and return all of them as a list

Specified by:

asList in interface DataSet

Returns:

List of DataSet objects, each with 1 example only

splitTestAndTrain

public SplitTestAndTrain splitTestAndTrain(int numHoldout,
                                           Random rng)

Splits a dataset in to test and train randomly. This will modify the dataset in place to shuffle it before splitting into test/train!

Specified by:

splitTestAndTrain in interface DataSet

Parameters:

numHoldout - the number to hold out for training

rng - Random Number Generator to use to shuffle the dataset

Returns:

the pair of datasets for the train test split

splitTestAndTrain

public SplitTestAndTrain splitTestAndTrain(int numHoldout)

Splits a dataset in to test and train

Specified by:

splitTestAndTrain in interface DataSet

Parameters:

numHoldout - the number to hold out for training

Returns:

the pair of datasets for the train test split

getLabels

public INDArray getLabels()

Returns the labels for the dataset

Specified by:

getLabels in interface DataSet

Returns:

the labels for the dataset

getLabelName

public String getLabelName(int idx)

Specified by:

getLabelName in interface DataSet

Parameters:

idx - the index to pullRows the string label value out of the list if it exists

Returns:

the label opName

getLabelNames

public List<String> getLabelNames(INDArray idxs)

Specified by:

getLabelNames in interface DataSet

Parameters:

idxs - list of index to pullRows the string label value out of the list if it exists

Returns:

the label opName

setLabels

public void setLabels(INDArray labels)

Specified by:

setLabels in interface DataSet

sortByLabel

public void sortByLabel()

Organizes the dataset to minimize sampling error while still allowing efficient batching.

Specified by:

sortByLabel in interface DataSet

addRow

public void addRow(DataSet d,
                   int i)

Specified by:

addRow in interface DataSet

exampleSums

public INDArray exampleSums()

Specified by:

exampleSums in interface DataSet

exampleMaxs

public INDArray exampleMaxs()

Specified by:

exampleMaxs in interface DataSet

exampleMeans

public INDArray exampleMeans()

Specified by:

exampleMeans in interface DataSet

sample

public DataSet sample(int numSamples)

Sample without replacement and a random rng

Specified by:

sample in interface DataSet

Parameters:

numSamples - the number of samples to getFromOrigin

Returns:

a sample data transform without replacement

sample

public DataSet sample(int numSamples,
                      Random rng)

Sample without replacement

Specified by:

sample in interface DataSet

Parameters:

numSamples - the number of samples to getFromOrigin

rng - the rng to use

Returns:

the sampled dataset without replacement

sample

public DataSet sample(int numSamples,
                      boolean withReplacement)

Sample a dataset numSamples times

Specified by:

sample in interface DataSet

Parameters:

numSamples - the number of samples to getFromOrigin

withReplacement - the rng to use

Returns:

the sampled dataset without replacement

sample

public DataSet sample(int numSamples,
                      Random rng,
                      boolean withReplacement)

Sample a dataset

Specified by:

sample in interface DataSet

Parameters:

numSamples - the number of samples to getFromOrigin

rng - the rng to use

withReplacement - whether to allow duplicates (only tracked by example row number)

Returns:

the sample dataset

roundToTheNearest

public void roundToTheNearest(int roundTo)

Specified by:

roundToTheNearest in interface DataSet

numOutcomes

public int numOutcomes()

Description copied from interface: DataSet

Returns the number of outcomes (size of the labels array for each example)

Specified by:

numOutcomes in interface DataSet

numExamples

public int numExamples()

Description copied from interface: DataSet

Number of examples in the DataSet

Specified by:

numExamples in interface DataSet

toString

public String toString()

Overrides:

toString in class Object

getLabelNames

@Deprecated
public List<String> getLabelNames()

Deprecated.

Gets the optional label names

Specified by:

getLabelNames in interface DataSet

Returns:

getLabelNamesList

public List<String> getLabelNamesList()

Gets the optional label names

Specified by:

getLabelNamesList in interface DataSet

Returns:

setLabelNames

public void setLabelNames(List<String> labelNames)

Sets the label names, will throw an exception if the passed in label names doesn't equal the number of outcomes

Specified by:

setLabelNames in interface DataSet

Parameters:

labelNames - the label names to use

getColumnNames

@Deprecated
public List<String> getColumnNames()

Deprecated.

Optional column names of the data transform, this is mainly used for interpeting what columns are in the dataset

Specified by:

getColumnNames in interface DataSet

Returns:

setColumnNames

@Deprecated
public void setColumnNames(List<String> columnNames)

Deprecated.

Sets the column names, will throw an exception if the column names don't match the number of columns

Specified by:

setColumnNames in interface DataSet

Parameters:

columnNames -

splitTestAndTrain

public SplitTestAndTrain splitTestAndTrain(double fractionTrain)

Description copied from interface: DataSet

SplitV the DataSet into two DataSets randomly

Specified by:

splitTestAndTrain in interface DataSet

Parameters:

fractionTrain - Fraction (in range 0 to 1) of examples to be returned in the training DataSet object

iterator

public Iterator<DataSet> iterator()

Specified by:

iterator in interface Iterable<DataSet>

Specified by:

iterator in interface DataSet

getFeaturesMaskArray

public INDArray getFeaturesMaskArray()

Description copied from interface: DataSet

Input mask array: a mask array for input, where each value is in {0,1} in order to specify whether an input is actually present or not. Typically used for situations such as RNNs with variable length inputs

Specified by:

getFeaturesMaskArray in interface DataSet

Returns:

Input mask array

setFeaturesMaskArray

public void setFeaturesMaskArray(INDArray featuresMask)

Description copied from interface: DataSet

Set the features mask array in this DataSet

Specified by:

setFeaturesMaskArray in interface DataSet

getLabelsMaskArray

public INDArray getLabelsMaskArray()

Description copied from interface: DataSet

Labels (output) mask array: a mask array for input, where each value is in {0,1} in order to specify whether an output is actually present or not. Typically used for situations such as RNNs with variable length inputs or many- to-one situations.

Specified by:

getLabelsMaskArray in interface DataSet

Returns:

Labels (output) mask array

setLabelsMaskArray

public void setLabelsMaskArray(INDArray labelsMask)

Description copied from interface: DataSet

Set the labels mask array in this data set

Specified by:

setLabelsMaskArray in interface DataSet

hasMaskArrays

public boolean hasMaskArrays()

Description copied from interface: DataSet

Whether the labels or input (features) mask arrays are present for this DataSet

Specified by:

hasMaskArrays in interface DataSet

equals

public boolean equals(Object o)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object

getMemoryFootprint

public long getMemoryFootprint()

This method returns memory used by this DataSet

Specified by:

getMemoryFootprint in interface DataSet

Returns:

migrate

public void migrate()

Description copied from interface: DataSet

This method migrates this DataSet into current Workspace (if any)

Specified by:

migrate in interface DataSet

detach

public void detach()

Description copied from interface: DataSet

This method detaches this DataSet from current Workspace (if any)

Specified by:

detach in interface DataSet

isEmpty

public boolean isEmpty()

Specified by:

isEmpty in interface DataSet

Returns:

true if the DataSet object is empty (no features, labels, or masks)

toMultiDataSet

public MultiDataSet toMultiDataSet()

Specified by:

toMultiDataSet in interface DataSet