org.nd4j.linalg.lossfunctions.impl

Class LossMixtureDensity

java.lang.Object

org.nd4j.autodiff.functions.DifferentialFunction

org.nd4j.linalg.lossfunctions.impl.LossMixtureDensity

All Implemented Interfaces:

Serializable, ILossFunction

public class LossMixtureDensity
extends DifferentialFunction
implements ILossFunction

This is a cost function associated with a mixture-density network. For background, this is inspired by Bishop's work pioneering the mixture density network. The essence of the idea is that the cost function attempts to model the output as if it were a mixture of gaussian probability densities. The network attempts to converge on a cost function which involves the negative log likelihood of the output fitting a set of data and estimates the "alpha" contribution to of each of the distributions the 'mu' (mean) and 'sigma' (standard deviation) of each of the distributions. For a full description of the technique, refer to Bishop's work. Bishop CM. Mixture density networks, Neural Computing Research Group Report: NCRG/94/004, Aston University, Birmingham, 1994 https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/bishop-ncrg-94-004.pdf There is no public constructor, please use the builder to create an approriate mixture loss function for the number of outputs and number of mixtures you would like to fit. Note that this means that the output layer must provide (labelWidth+2)*mixtures output values in order to describe the distributions of vectors of width labelWidth. Please ensure that the size of the output layer matches the number of mixtures provided.

Author:

Jonathan Arney

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	LossMixtureDensity.Builder
static class	LossMixtureDensity.MixtureDensityComponents This class is a data holder for the mixture density components for convenient manipulation.

Field Summary

Fields inherited from class org.nd4j.autodiff.functions.DifferentialFunction

dimensions, extraArgs, inPlace, sameDiff, scalarValue

Constructor Summary

Constructors

Constructor and Description
LossMixtureDensity()

Method Summary

Modifier and Type	Method and Description
static LossMixtureDensity.Builder	builder()
INDArray	computeGradient(INDArray labels, INDArray preOutput, IActivation activationFn, INDArray mask) This method returns the gradient of the cost function with respect to the output from the previous layer.
org.nd4j.linalg.primitives.Pair<Double,INDArray>	computeGradientAndScore(INDArray labels, INDArray preOutput, IActivation activationFn, INDArray mask, boolean average) Compute both the score (loss function value) and gradient.
double	computeScore(INDArray labels, INDArray preOutput, IActivation activationFn, INDArray mask, boolean average) Computes the aggregate score as a sum of all of the individual scores of each of the labels against each of the outputs of the network.
INDArray	computeScoreArray(INDArray labels, INDArray preOutput, IActivation activationFn, INDArray mask) This method returns the score for each of the given outputs against the given set of labels.
List<SDVariable>	doDiff(List<SDVariable> f1) The actual implementation for automatic differentiation.
LossMixtureDensity.MixtureDensityComponents	extractComponents(INDArray output)
int	getLabelWidth() Returns the width of each label vector.
int	getNMixtures() Returns the number of gaussians this loss function will attempt to find.
void	initFromOnnx(OnnxProto3.NodeProto node, SameDiff initWith, Map<String,OnnxProto3.AttributeProto> attributesForNode, OnnxProto3.GraphProto graph) Iniitialize the function from the given OnnxProto3.NodeProto
void	initFromTensorFlow(NodeDef nodeDef, SameDiff initWith, Map<String,AttrValue> attributesForNode, GraphDef graph) Initialize the function from the given NodeDef
String	name() The opName of this function
String	onnxName() The opName of this function in onnx
String	opName() The name of the op
Op.Type	opType() The type of the op
SDVariable[]	outputVariables() Return the output variables for this differential function.
SDVariable[]	outputVariables(String baseName) Return the output functions for this differential function.
String	tensorflowName() The opName of this function tensorflow
String	toString()

Methods inherited from class org.nd4j.autodiff.functions.DifferentialFunction

arg, args, asProperties, attributeAdaptersForFunction, calculateOutputShape, configFieldName, diff, dup, equals, f, getValue, hashCode, hasPlaceHolderInputs, isConfigProperties, larg, mappingsForFunction, onnxNames, opNum, propertiesForFunction, rarg, resolvePropertiesFromSameDiffBeforeExecution, setInstanceId, setValueFor, tensorflowNames

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

LossMixtureDensity

public LossMixtureDensity()

Method Detail

extractComponents

public LossMixtureDensity.MixtureDensityComponents extractComponents(INDArray output)

computeScore

public double computeScore(INDArray labels,
                           INDArray preOutput,
                           IActivation activationFn,
                           INDArray mask,
                           boolean average)

Computes the aggregate score as a sum of all of the individual scores of each of the labels against each of the outputs of the network. For the mixture density network, this is the negative log likelihood that the given labels fall within the probability distribution described by the mixture of gaussians of the network output.

Specified by:

computeScore in interface ILossFunction

Parameters:

labels - Labels to score against the network.

preOutput - Output of the network (before activation function has been called).

activationFn - Activation function for the network.

mask - Mask to be applied to labels (not used for MDN).

average - Whether or not to return an average instead of a total score (not used).

Returns:

Returns a single double which corresponds to the total score of all label values.

computeScoreArray

public INDArray computeScoreArray(INDArray labels,
                                  INDArray preOutput,
                                  IActivation activationFn,
                                  INDArray mask)

This method returns the score for each of the given outputs against the given set of labels. For a mixture density network, this is done by extracting the "alpha", "mu", and "sigma" components of each gaussian and computing the negative log likelihood that the labels fall within a linear combination of these gaussian distributions. The smaller the negative log likelihood, the higher the probability that the given labels actually would fall within the distribution. Therefore by minimizing the negative log likelihood, we get to a position of highest probability that the gaussian mixture explains the phenomenon.

Specified by:

computeScoreArray in interface ILossFunction

Parameters:

labels - Labels give the sample output that the network should be trying to converge on.

preOutput - The output of the last layer (before applying the activation function).

activationFn - The activation function of the current layer.

mask - Mask to apply to score evaluation (not supported for this cost function).

Returns:

computeGradient

public INDArray computeGradient(INDArray labels,
                                INDArray preOutput,
                                IActivation activationFn,
                                INDArray mask)

This method returns the gradient of the cost function with respect to the output from the previous layer. For this cost function, the gradient is derived from Bishop's paper "Mixture Density Networks" (1994) which gives an elegant closed-form expression for the derivatives with respect to each of the output components.

Specified by:

computeGradient in interface ILossFunction

Parameters:

labels - Labels to train on.

preOutput - Output of neural network before applying the final activation function.

activationFn - Activation function of output layer.

mask - Mask to apply to gradients.

Returns:

Gradient of cost function with respect to preOutput parameters.

computeGradientAndScore

public org.nd4j.linalg.primitives.Pair<Double,INDArray> computeGradientAndScore(INDArray labels,
                                                                                INDArray preOutput,
                                                                                IActivation activationFn,
                                                                                INDArray mask,
                                                                                boolean average)

Description copied from interface: ILossFunction

Compute both the score (loss function value) and gradient. This is equivalent to calling ILossFunction.computeScore(INDArray, INDArray, IActivation, INDArray, boolean) and ILossFunction.computeGradient(INDArray, INDArray, IActivation, INDArray) individually

Specified by:

computeGradientAndScore in interface ILossFunction

Parameters:

labels - Label/expected output

preOutput - Output of the model (neural network)

activationFn - Activation function that should be applied to preOutput

mask - Mask array; may be null

average - Whether the score should be averaged (divided by number of rows in labels/output) or not

Returns:

The score (loss function value) and gradient

name

public String name()

The opName of this function

Specified by:

name in interface ILossFunction

Returns:

getNMixtures

public int getNMixtures()

Returns the number of gaussians this loss function will attempt to find.

Returns:

Number of gaussians to find.

getLabelWidth

public int getLabelWidth()

Returns the width of each label vector.

Returns:

Width of label vectors expected.

toString

public String toString()

Overrides:

toString in class Object

builder

public static LossMixtureDensity.Builder builder()

outputVariables

public SDVariable[] outputVariables()

Description copied from class: DifferentialFunction

Return the output variables for this differential function. Note that this op *may* dynamically generate variable outputs.

Overrides:

outputVariables in class DifferentialFunction

Returns:

outputVariables

public SDVariable[] outputVariables(String baseName)

Description copied from class: DifferentialFunction

Return the output functions for this differential function.

Specified by:

outputVariables in class DifferentialFunction

Returns:

doDiff

public List<SDVariable> doDiff(List<SDVariable> f1)

Description copied from class: DifferentialFunction

The actual implementation for automatic differentiation.

Specified by:

doDiff in class DifferentialFunction

Returns:

opName

public String opName()

Description copied from class: DifferentialFunction

The name of the op

Overrides:

opName in class DifferentialFunction

Returns:

opType

public Op.Type opType()

Description copied from class: DifferentialFunction

The type of the op

Overrides:

opType in class DifferentialFunction

Returns:

initFromTensorFlow

public void initFromTensorFlow(NodeDef nodeDef,
                               SameDiff initWith,
                               Map<String,AttrValue> attributesForNode,
                               GraphDef graph)

Description copied from class: DifferentialFunction

Initialize the function from the given NodeDef

Specified by:

initFromTensorFlow in class DifferentialFunction

initFromOnnx

public void initFromOnnx(OnnxProto3.NodeProto node,
                         SameDiff initWith,
                         Map<String,OnnxProto3.AttributeProto> attributesForNode,
                         OnnxProto3.GraphProto graph)

Description copied from class: DifferentialFunction

Iniitialize the function from the given OnnxProto3.NodeProto

Specified by:

initFromOnnx in class DifferentialFunction

onnxName

public String onnxName()

Description copied from class: DifferentialFunction

The opName of this function in onnx

Specified by:

onnxName in class DifferentialFunction

Returns:

tensorflowName

public String tensorflowName()

Description copied from class: DifferentialFunction

The opName of this function tensorflow

Specified by:

tensorflowName in class DifferentialFunction

Returns: