org.nd4j.linalg.dataset.api.preprocessor

Class MultiNormalizerHybrid

java.lang.Object

org.nd4j.linalg.dataset.api.preprocessor.AbstractNormalizer

org.nd4j.linalg.dataset.api.preprocessor.MultiNormalizerHybrid

All Implemented Interfaces:

Serializable, MultiDataSetPreProcessor, MultiDataNormalization, Normalizer<MultiDataSet>

public class MultiNormalizerHybrid
extends AbstractNormalizer
implements MultiDataNormalization, Serializable

Pre processor for MultiDataSet that can be configured to use different normalization strategies for different inputs and outputs, or none at all. Can be used for example when one input should be normalized, but a different one should be untouched because it's the input for an embedding layer. Alternatively, one might want to mix standardization and min-max scaling for different inputs and outputs.

By default, no normalization is applied. There are methods to configure the desired normalization strategy for inputs and outputs either globally or on an individual input/output level. Specific input/output strategies will override global ones.

Author:

Ede Meijer

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
MultiNormalizerHybrid()

Method Summary

Modifier and Type	Method and Description
void	fit(MultiDataSet dataSet) Fit a MultiDataSet (only compute based on the statistics from this dataset)
void	fit(MultiDataSetIterator iterator) Iterates over a dataset accumulating statistics for normalization
Map<Integer,NormalizerStats>	getInputStats() Get the map of normalization statistics per input
NormalizerStats	getInputStats(int input) Get normalization statistics for a given input.
Map<Integer,NormalizerStats>	getOutputStats() Get the map of normalization statistics per output
NormalizerStats	getOutputStats(int output) Get normalization statistics for a given output.
NormalizerType	getType() Get the enum opType of this normalizer
protected boolean	isFit()
MultiNormalizerHybrid	minMaxScaleAllInputs() Apply min-max scaling to all inputs, except the ones individually configured
MultiNormalizerHybrid	minMaxScaleAllInputs(double rangeFrom, double rangeTo) Apply min-max scaling to all inputs, except the ones individually configured
MultiNormalizerHybrid	minMaxScaleAllOutputs() Apply min-max scaling to all outputs, except the ones individually configured
MultiNormalizerHybrid	minMaxScaleAllOutputs(double rangeFrom, double rangeTo) Apply min-max scaling to all outputs, except the ones individually configured
MultiNormalizerHybrid	minMaxScaleInput(int input) Apply min-max scaling to a specific input, overriding the global input strategy if any
MultiNormalizerHybrid	minMaxScaleInput(int input, double rangeFrom, double rangeTo) Apply min-max scaling to a specific input, overriding the global input strategy if any
MultiNormalizerHybrid	minMaxScaleOutput(int output) Apply min-max scaling to a specific output, overriding the global output strategy if any
MultiNormalizerHybrid	minMaxScaleOutput(int output, double rangeFrom, double rangeTo) Apply min-max scaling to a specific output, overriding the global output strategy if any
void	preProcess(MultiDataSet data) Preprocess the MultiDataSet
void	revert(MultiDataSet data) Undo (revert) the normalization applied by this DataNormalization instance (arrays are modified in-place)
void	revertFeatures(INDArray[] features) Undo (revert) the normalization applied by this DataNormalization instance to the entire inputs array
void	revertFeatures(INDArray[] features, INDArray[] maskArrays) Undo (revert) the normalization applied by this DataNormalization instance to the entire inputs array
void	revertFeatures(INDArray[] features, INDArray[] maskArrays, int input) Undo (revert) the normalization applied by this DataNormalization instance to the features of a particular input
void	revertLabels(INDArray[] labels) Undo (revert) the normalization applied by this DataNormalization instance to the entire outputs array
void	revertLabels(INDArray[] labels, INDArray[] maskArrays) Undo (revert) the normalization applied by this DataNormalization instance to the entire outputs array
void	revertLabels(INDArray[] labels, INDArray[] maskArrays, int output) Undo (revert) the normalization applied by this DataNormalization instance to the labels of a particular output
MultiNormalizerHybrid	standardizeAllInputs() Apply standardization to all inputs, except the ones individually configured
MultiNormalizerHybrid	standardizeAllOutputs() Apply standardization to all outputs, except the ones individually configured
MultiNormalizerHybrid	standardizeInput(int input) Apply standardization to a specific input, overriding the global input strategy if any
MultiNormalizerHybrid	standardizeOutput(int output) Apply standardization to a specific output, overriding the global output strategy if any
void	transform(MultiDataSet data) Transform the dataset

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MultiNormalizerHybrid

public MultiNormalizerHybrid()

Method Detail

standardizeAllInputs

public MultiNormalizerHybrid standardizeAllInputs()

Apply standardization to all inputs, except the ones individually configured

Returns:

the normalizer

minMaxScaleAllInputs

public MultiNormalizerHybrid minMaxScaleAllInputs()

Apply min-max scaling to all inputs, except the ones individually configured

Returns:

the normalizer

minMaxScaleAllInputs

public MultiNormalizerHybrid minMaxScaleAllInputs(double rangeFrom,
                                                  double rangeTo)

Apply min-max scaling to all inputs, except the ones individually configured

Parameters:

rangeFrom - lower bound of the target range

rangeTo - upper bound of the target range

Returns:

the normalizer

standardizeInput

public MultiNormalizerHybrid standardizeInput(int input)

Apply standardization to a specific input, overriding the global input strategy if any

Parameters:

input - the index of the input

Returns:

the normalizer

minMaxScaleInput

public MultiNormalizerHybrid minMaxScaleInput(int input)

Apply min-max scaling to a specific input, overriding the global input strategy if any

Parameters:

input - the index of the input

Returns:

the normalizer

minMaxScaleInput

public MultiNormalizerHybrid minMaxScaleInput(int input,
                                              double rangeFrom,
                                              double rangeTo)

Apply min-max scaling to a specific input, overriding the global input strategy if any

Parameters:

input - the index of the input

rangeFrom - lower bound of the target range

rangeTo - upper bound of the target range

Returns:

the normalizer

standardizeAllOutputs

public MultiNormalizerHybrid standardizeAllOutputs()

Apply standardization to all outputs, except the ones individually configured

Returns:

the normalizer

minMaxScaleAllOutputs

public MultiNormalizerHybrid minMaxScaleAllOutputs()

Apply min-max scaling to all outputs, except the ones individually configured

Returns:

the normalizer

minMaxScaleAllOutputs

public MultiNormalizerHybrid minMaxScaleAllOutputs(double rangeFrom,
                                                   double rangeTo)

Apply min-max scaling to all outputs, except the ones individually configured

Parameters:

rangeFrom - lower bound of the target range

rangeTo - upper bound of the target range

Returns:

the normalizer

standardizeOutput

public MultiNormalizerHybrid standardizeOutput(int output)

Apply standardization to a specific output, overriding the global output strategy if any

Parameters:

output - the index of the input

Returns:

the normalizer

minMaxScaleOutput

public MultiNormalizerHybrid minMaxScaleOutput(int output)

Apply min-max scaling to a specific output, overriding the global output strategy if any

Parameters:

output - the index of the input

Returns:

the normalizer

minMaxScaleOutput

public MultiNormalizerHybrid minMaxScaleOutput(int output,
                                               double rangeFrom,
                                               double rangeTo)

Apply min-max scaling to a specific output, overriding the global output strategy if any

Parameters:

output - the index of the input

rangeFrom - lower bound of the target range

rangeTo - upper bound of the target range

Returns:

the normalizer

getInputStats

public NormalizerStats getInputStats(int input)

Get normalization statistics for a given input.

Parameters:

input - the index of the input

Returns:

implementation of NormalizerStats corresponding to the normalization strategy selected

getOutputStats

public NormalizerStats getOutputStats(int output)

Get normalization statistics for a given output.

Parameters:

output - the index of the output

Returns:

implementation of NormalizerStats corresponding to the normalization strategy selected

getInputStats

public Map<Integer,NormalizerStats> getInputStats()

Get the map of normalization statistics per input

Returns:

map of input indices pointing to NormalizerStats instances

getOutputStats

public Map<Integer,NormalizerStats> getOutputStats()

Get the map of normalization statistics per output

Returns:

map of output indices pointing to NormalizerStats instances

fit

public void fit(@NonNull
                MultiDataSet dataSet)

Fit a MultiDataSet (only compute based on the statistics from this dataset)

Specified by:

fit in interface Normalizer<MultiDataSet>

Parameters:

dataSet - the dataset to compute on

fit

public void fit(@NonNull
                MultiDataSetIterator iterator)

Iterates over a dataset accumulating statistics for normalization

Specified by:

fit in interface MultiDataNormalization

Parameters:

iterator - the iterator to use for collecting statistics

transform

public void transform(@NonNull
                      MultiDataSet data)

Transform the dataset

Specified by:

transform in interface Normalizer<MultiDataSet>

Parameters:

data - the dataset to pre process

preProcess

public void preProcess(@NonNull
                       MultiDataSet data)

Description copied from interface: MultiDataSetPreProcessor

Preprocess the MultiDataSet

Specified by:

preProcess in interface MultiDataSetPreProcessor

Specified by:

preProcess in interface MultiDataNormalization

revert

public void revert(@NonNull
                   MultiDataSet data)

Undo (revert) the normalization applied by this DataNormalization instance (arrays are modified in-place)

Specified by:

revert in interface Normalizer<MultiDataSet>

Parameters:

data - MultiDataSet to revert the normalization on

getType

public NormalizerType getType()

Description copied from interface: Normalizer

Get the enum opType of this normalizer

Specified by:

getType in interface Normalizer<MultiDataSet>

Returns:

the opType

See Also:

NormalizerSerializerStrategy.getSupportedType()

revertFeatures

public void revertFeatures(@NonNull
                           INDArray[] features)

Undo (revert) the normalization applied by this DataNormalization instance to the entire inputs array

Specified by:

revertFeatures in interface MultiDataNormalization

Parameters:

features - The normalized array of inputs

revertFeatures

public void revertFeatures(@NonNull
                           INDArray[] features,
                           INDArray[] maskArrays)

Undo (revert) the normalization applied by this DataNormalization instance to the entire inputs array

Specified by:

revertFeatures in interface MultiDataNormalization

Parameters:

features - The normalized array of inputs

maskArrays - Optional mask arrays belonging to the inputs

revertFeatures

public void revertFeatures(@NonNull
                           INDArray[] features,
                           INDArray[] maskArrays,
                           int input)

Undo (revert) the normalization applied by this DataNormalization instance to the features of a particular input

Parameters:

features - The normalized array of inputs

maskArrays - Optional mask arrays belonging to the inputs

input - the index of the input to revert normalization on

revertLabels

public void revertLabels(@NonNull
                         INDArray[] labels)

Undo (revert) the normalization applied by this DataNormalization instance to the entire outputs array

Specified by:

revertLabels in interface MultiDataNormalization

Parameters:

labels - The normalized array of outputs

revertLabels

public void revertLabels(@NonNull
                         INDArray[] labels,
                         INDArray[] maskArrays)

Undo (revert) the normalization applied by this DataNormalization instance to the entire outputs array

Specified by:

revertLabels in interface MultiDataNormalization

Parameters:

labels - The normalized array of outputs

maskArrays - Optional mask arrays belonging to the outputs

revertLabels

public void revertLabels(@NonNull
                         INDArray[] labels,
                         INDArray[] maskArrays,
                         int output)

Undo (revert) the normalization applied by this DataNormalization instance to the labels of a particular output

Parameters:

labels - The normalized array of outputs

maskArrays - Optional mask arrays belonging to the outputs

output - the index of the output to revert normalization on

isFit

protected boolean isFit()

Specified by:

isFit in class AbstractNormalizer