<p>I'm new to neural networks and I try to make a model that is guessing if a point is below or above relative to a function output. The idea is inspired from this video <a href="https://youtu.be/DGxIcDjPzac" rel="nofollow noreferrer">https://youtu.be/DGxIcDjPzac</a> .</p>
<p>What am I doing wrong?</p>
<p>In the gif below I start the training but it seems that is not working. The blue line is the function (y = x + 50) and all the points above it should be green, but aren't. In order to simplify the example and to debug easier, I picked a simple function such that I can use only a perception for the model.</p>
<p>I also made a method <code>backPropagationDebug(...)</code> to display for the points that are predicted wrong all that matrices in each step, but I couldn't find what's wrong.</p>
<p><a href="https://i.sstatic.net/qqOcg.gif" rel="nofollow noreferrer"><img src="https://i.sstatic.net/qqOcg.gif" alt="enter image description here" /></a></p>
<pre><code>public void backPropagation(double[][] input, double[][] expected) {
    double[][][] outputs = getOutputs(input);

    double[][] currentOutput = outputs[outputs.length - 1];
    double[][] currentError = Matrix.subtract(expected, currentOutput);

    for (int i = brain.length - 1; i &gt;= 0; i--) {
        final double[][] layer = brain[i];
        final double[][] previousOutput = outputs[i];

        final double[][] layerTranspose = Matrix.transpose(layer);
        final double[][] previousError = Matrix.multiply(layerTranspose, currentError);

        /* FIST BIT */
        double[][] errorSigmoid = Matrix.copyOf(currentError);

        for (int k = 0; k &lt; errorSigmoid.length; k++) {
            errorSigmoid[k][0] *= - derivativeActivationFunction(currentOutput[k][0]);
        }

        /* SECOND BIT */
        final double[][] slopeMatrix = Matrix.multiply(errorSigmoid, Matrix.transpose(previousOutput));

        /* UPDATE THE WEIGHTS */
        for (int k = 0; k &lt; layer.length; k++) {
            for (int l = 0; l &lt; layer[0].length; l++) {
                layer[k][l] = layer[k][l] - learningRate * slopeMatrix[k][l];
            }
        }

        currentOutput = previousOutput;
        currentError = previousError;
    }
}
</code></pre>
<p>The backpropagation steps are inspired from this formulas:
<a href="https://i.sstatic.net/J8O5L.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/J8O5L.png" alt="enter image description here" /></a>
<a href="https://i.sstatic.net/qD1vy.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/qD1vy.png" alt="enter image description here" /></a></p>
<p>(From: Make Your Own Neural Network By Tariq Rashid)</p>
<p>The code is on github: <a href="https://github.com/StamateValentin/Artificial-Intelligence-Playground/tree/7a7446b7faedd7673bc53a62304ff3a5180d77eb" rel="nofollow noreferrer">https://github.com/StamateValentin/Artificial-Intelligence-Playground/tree/7a7446b7faedd7673bc53a62304ff3a5180d77eb</a></p>
<p>The resources I used are in the README.md file.</p>

CC BY-SA 4.0
2021-06-15T11:51:06.340Z
Please include any mathematics using LaTeX; just pasting images of mathematics is frowned upon.
The Pointer
CC BY-SA 4.0
2021-06-15T17:18:33.527Z
Welcome to the site. What is your specific question? Please put it somewhere it can be recognized easily.
S2673
CC BY-SA 4.0
2021-06-15T17:50:45.100Z
I added the question.
Valentin Stamate
CC BY-SA 4.0
2021-06-15T18:12:00.257Z
@Valentin, Thank you, that helps. Also, if you don’t put @[my username] at the beginning of your comment, then I won’t be notified that you responded. [(More information about the comment system)](https://ai.stackexchange.com/help/privileges/comment)
S2673
<neural-networks><backpropagation><supervised-learning><perceptron>
Backpropagation not working as expected
Valentin Stamate
# Backpropagation not working as expected

(Asked by: username_0 on 2021-06-15)

I'm new to neural networks and I try to make a model that is guessing if a point is below or above relative to a function output. The idea is inspired from this video https://youtu.be/DGxIcDjPzac .
What am I doing wrong?
In the gif below I start the training but it seems that is not working. The blue line is the function (y = x + 50) and all the points above it should be green, but aren't. In order to simplify the example and to debug easier, I picked a simple function such that I can use only a perception for the model.
I also made a method backPropagationDebug(...) to display for the points that are predicted wrong all that matrices in each step, but I couldn't find what's wrong.

public void backPropagation(double[][] input, double[][] expected) {
    double[][][] outputs = getOutputs(input);

    double[][] currentOutput = outputs[outputs.length - 1];
    double[][] currentError = Matrix.subtract(expected, currentOutput);

    for (int i = brain.length - 1; i >= 0; i--) {
        final double[][] layer = brain[i];
        final double[][] previousOutput = outputs[i];

        final double[][] layerTranspose = Matrix.transpose(layer);
        final double[][] previousError = Matrix.multiply(layerTranspose, currentError);

        /* FIST BIT */
        double[][] errorSigmoid = Matrix.copyOf(currentError);

        for (int k = 0; k < errorSigmoid.length; k++) {
            errorSigmoid[k][0] *= - derivativeActivationFunction(currentOutput[k][0]);
        }

        /* SECOND BIT */
        final double[][] slopeMatrix = Matrix.multiply(errorSigmoid, Matrix.transpose(previousOutput));

        /* UPDATE THE WEIGHTS */
        for (int k = 0; k < layer.length; k++) {
            for (int l = 0; l < layer[0].length; l++) {
                layer[k][l] = layer[k][l] - learningRate * slopeMatrix[k][l];
            }
        }

        currentOutput = previousOutput;
        currentError = previousError;
    }
}

The backpropagation steps are inspired from this formulas:


(From: Make Your Own Neural Network By Tariq Rashid)
The code is on github: https://github.com/StamateValentin/Artificial-Intelligence-Playground/tree/7a7446b7faedd7673bc53a62304ff3a5180d77eb
The resources I used are in the README.md file.

**Tags:** neural-networks, backpropagation, supervised-learning, perceptron

> **Comments:**
> **username_1**: Please include any mathematics using LaTeX; just pasting images of mathematics is frowned upon.  
> **username_2**: Welcome to the site. What is your specific question? Please put it somewhere it can be recognized easily.  
> **username_0**: I added the question.  
> **username_2**: @Valentin, Thank you, that helps. Also, if you don’t put @[my username] at the beginning of your comment, then I won’t be notified that you responded. [(More information about the comment system)](https://ai.stackexchange.com/help/privileges/comment)  

---

## 0 answers

