Body: <p>I'm new to neural networks and I try to make a model that is guessing if a point is below or above relative to a function output. The idea is inspired from this video <a href="https://youtu.be/DGxIcDjPzac" rel="nofollow noreferrer">https://youtu.be/DGxIcDjPzac</a> .</p>
  <p>What am I doing wrong?</p>
  <p>In the gif below I start the training but it seems that is not working. The blue line is the function (y = x + 50) and all the points above it should be green, but aren't. In order to simplify the example and to debug easier, I picked a simple function such that I can use only a perception for the model.</p>
  <p>I also made a method <code>backPropagationDebug(...)</code> to display for the points that are predicted wrong all that matrices in each step, but I couldn't find what's wrong.</p>
  <p><a href="https://i.sstatic.net/qqOcg.gif" rel="nofollow noreferrer"><img src="https://i.sstatic.net/qqOcg.gif" alt="enter image description here" /></a></p>
  <pre><code>public void backPropagation(double[][] input, double[][] expected) {
      double[][][] outputs = getOutputs(input);
  
      double[][] currentOutput = outputs[outputs.length - 1];
      double[][] currentError = Matrix.subtract(expected, currentOutput);
  
      for (int i = brain.length - 1; i &gt;= 0; i--) {
          final double[][] layer = brain[i];
          final double[][] previousOutput = outputs[i];
  
          final double[][] layerTranspose = Matrix.transpose(layer);
          final double[][] previousError = Matrix.multiply(layerTranspose, currentError);
  
          /* FIST BIT */
          double[][] errorSigmoid = Matrix.copyOf(currentError);
  
          for (int k = 0; k &lt; errorSigmoid.length; k++) {
              errorSigmoid[k][0] *= - derivativeActivationFunction(currentOutput[k][0]);
          }
  
          /* SECOND BIT */
          final double[][] slopeMatrix = Matrix.multiply(errorSigmoid, Matrix.transpose(previousOutput));
  
          /* UPDATE THE WEIGHTS */
          for (int k = 0; k &lt; layer.length; k++) {
              for (int l = 0; l &lt; layer[0].length; l++) {
                  layer[k][l] = layer[k][l] - learningRate * slopeMatrix[k][l];
              }
          }
  
          currentOutput = previousOutput;
          currentError = previousError;
      }
  }
  </code></pre>
  <p>The backpropagation steps are inspired from this formulas:
  <a href="https://i.sstatic.net/J8O5L.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/J8O5L.png" alt="enter image description here" /></a>
  <a href="https://i.sstatic.net/qD1vy.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/qD1vy.png" alt="enter image description here" /></a></p>
  <p>(From: Make Your Own Neural Network By Tariq Rashid)</p>
  <p>The code is on github: <a href="https://github.com/StamateValentin/Artificial-Intelligence-Playground/tree/7a7446b7faedd7673bc53a62304ff3a5180d77eb" rel="nofollow noreferrer">https://github.com/StamateValentin/Artificial-Intelligence-Playground/tree/7a7446b7faedd7673bc53a62304ff3a5180d77eb</a></p>
  <p>The resources I used are in the README.md file.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-15T11:51:06.340Z
Comments.Text: Please include any mathematics using LaTeX; just pasting images of mathematics is frowned upon.
Comments.UserDisplayName: The Pointer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-15T17:18:33.527Z
Comments.Text: Welcome to the site. What is your specific question? Please put it somewhere it can be recognized easily.
Comments.UserDisplayName: S2673
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-15T17:50:45.100Z
Comments.Text: I added the question.
Comments.UserDisplayName: Valentin Stamate
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-15T18:12:00.257Z
Comments.Text: @Valentin, Thank you, that helps. Also, if you don’t put @[my username] at the beginning of your comment, then I won’t be notified that you responded. [(More information about the comment system)](https://ai.stackexchange.com/help/privileges/comment)
Comments.UserDisplayName: S2673
Tags: <neural-networks><backpropagation><supervised-learning><perceptron>
Title: Backpropagation not working as expected
UserDisplayName: Valentin Stamate
ThreadText: # Backpropagation not working as expected
  
  (Asked by: username_0 on 2021-06-15)
  
  I'm new to neural networks and I try to make a model that is guessing if a point is below or above relative to a function output. The idea is inspired from this video <https://youtu.be/DGxIcDjPzac> .
  
  What am I doing wrong?
  
  In the gif below I start the training but it seems that is not working. The blue line is the function (y = x + 50) and all the points above it should be green, but aren't. In order to simplify the example and to debug easier, I picked a simple function such that I can use only a perception for the model.
  
  I also made a method `backPropagationDebug(...)` to display for the points that are predicted wrong all that matrices in each step, but I couldn't find what's wrong.
  
  [![enter image description here](https://i.sstatic.net/qqOcg.gif)](https://i.sstatic.net/qqOcg.gif)
  
  ```
  public void backPropagation(double[][] input, double[][] expected) {
      double[][][] outputs = getOutputs(input);
  
      double[][] currentOutput = outputs[outputs.length - 1];
      double[][] currentError = Matrix.subtract(expected, currentOutput);
  
      for (int i = brain.length - 1; i >= 0; i--) {
          final double[][] layer = brain[i];
          final double[][] previousOutput = outputs[i];
  
          final double[][] layerTranspose = Matrix.transpose(layer);
          final double[][] previousError = Matrix.multiply(layerTranspose, currentError);
  
          /* FIST BIT */
          double[][] errorSigmoid = Matrix.copyOf(currentError);
  
          for (int k = 0; k < errorSigmoid.length; k++) {
              errorSigmoid[k][0] *= - derivativeActivationFunction(currentOutput[k][0]);
          }
  
          /* SECOND BIT */
          final double[][] slopeMatrix = Matrix.multiply(errorSigmoid, Matrix.transpose(previousOutput));
  
          /* UPDATE THE WEIGHTS */
          for (int k = 0; k < layer.length; k++) {
              for (int l = 0; l < layer[0].length; l++) {
                  layer[k][l] = layer[k][l] - learningRate * slopeMatrix[k][l];
              }
          }
  
          currentOutput = previousOutput;
          currentError = previousError;
      }
  }
  
  ```
  
  The backpropagation steps are inspired from this formulas:
  [![enter image description here](https://i.sstatic.net/J8O5L.png)](https://i.sstatic.net/J8O5L.png)
  [![enter image description here](https://i.sstatic.net/qD1vy.png)](https://i.sstatic.net/qD1vy.png)
  
  (From: Make Your Own Neural Network By Tariq Rashid)
  
  The code is on github: <https://github.com/StamateValentin/Artificial-Intelligence-Playground/tree/7a7446b7faedd7673bc53a62304ff3a5180d77eb>
  
  The resources I used are in the README.md file.
  
  **Tags:** neural-networks, backpropagation, supervised-learning, perceptron
  
  > **Comments:**
  > **username_1**: Please include any mathematics using LaTeX; just pasting images of mathematics is frowned upon.  
  > **username_2**: Welcome to the site. What is your specific question? Please put it somewhere it can be recognized easily.  
  > **username_0**: I added the question.  
  > **username_2**: @username_0, Thank you, that helps. Also, if you don’t put @[my username] at the beginning of your comment, then I won’t be notified that you responded. [(More information about the comment system)](https://ai.stackexchange.com/help/privileges/comment)  
  
  ---
  
  ## 0 answers
  
  