Answers.Body: <p>I believe the best way to do this is using <strong>numerical gradient</strong>. To understand the concept, we need to look the definition of derivatives using limits:</p>
  
  <p><a href="https://i.sstatic.net/9Qanu.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/9Qanu.png" alt="enter image description here"></a></p>
  
  <p>It means that, when you don't know how to derive some formula (or you just don't want to), you can approximate it by computing the output for a small change in input, subtract from the original result (no change), and normalize by this change.</p>
  
  <p>Example:
  We know the derivative of f(x) = x^2 is f'(x) = 2x. But, let suppose we don't and we are using x = 3 and h = 0.001 (it tends to zero in fact):</p>
  
  <p>f(3 + 0.001) = (3 + 0.001)^2 = 9,006 (approximately) <br>
  f(3) = 3^2 = 9</p>
  
  <p>Thus,</p>
  
  <p>(9,006 - 9) / 0.001 = 6</p>
  
  <p>It's is exactly to f'(3) = 2*3 = 6.</p>
  
  <p>In practice, if you want to know if your backpropagation is correct, </p>
  
  <ol>
  <li>pass a single example (x1) through your network and computes the output (o1). Compute the gradient with respect to x1 (d1).</li>
  <li>Then, add a small value to the input (h), pass through the network again, and computes the new output (o2).</li>
  <li>Subtract o2 - o1 and divide by h. It must be close enough to d1.</li>
  </ol>
  
  <p>That's it. It's called <strong>gradient checking</strong>. I hope it helps.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-02-28T16:43:56.067Z
Answers.LastActivityDate: 2018-03-03T13:25:55.020Z
Answers.LastEditDate: 2018-03-03T13:25:55.020Z
Answers.UserDisplayName: Arnaldo Gualberto
Answers.Body: <p>One thing that you should know is that the sigmoid function limits the output to a value between 0 and 1, which means that using a lot of hidden layers will lead quickly to a Vanishing Gradient.</p>
  
  <p>Try to use relu activation function, it has the property to output all the information it gets from the previous layer.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-06-28T17:24:46.063Z
Answers.LastActivityDate: 2017-06-28T17:24:46.063Z
Answers.UserDisplayName: Amar ALI-BEY
Answers.Body: <p>Actually the implementation was correct, </p>
  
  <p>The source of the problem that causes a big error and really slow learning was the architecture of the neural network it self, the ANN has 7 hidden layers which causes the vanishing gradient problem.</p>
  
  <p>When I have decreased the ANN layers to 3 the cost of error was widely reduced besides of that the learning process was faster.</p>
  
  <p>Another common solution is to use RELU or ELU or SELU in the hidden layers of the neural network instead of the sigmoid function</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-27T07:41:57.313Z
Answers.Text: if you change the activation function to relu or elu that would fix the issue
Answers.UserDisplayName: Anas EL KORCHI
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-05T07:52:37.667Z
Answers.Text: Yet another common solution is residual network, i.e., adding fixed weight connections skipping layers. I like it most as it works without modifying the structure / activation function.
Answers.UserDisplayName: maaartinus
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-01-27T22:54:01.730Z
Answers.LastActivityDate: 2018-01-27T13:37:15.057Z
Answers.LastEditDate: 2018-01-27T13:37:15.057Z
Answers.UserDisplayName: Anas EL KORCHI
Body: <p>I am working on an implementation of the back propagation algorithm. What I have implemented so far seems working but I can't be sure that the algorithm is well implemented, here is what I have noticed during training test of my network:</p>
  
  <p>Specification of the implementation:</p>
  
  <ul>
  <li>A data set containing almost 100000 raw containing (3 variable as input, the sinus of the sum of those three variables as expected output).</li>
  <li>The network does have 7 layers, all the layers use the sigmoid activation function</li>
  </ul>
  
  <p>When I run the back propagation training process:</p>
  
  <ul>
  <li>The minimum of costs of the error is found at the fourth iteration (<strong>The minimum cost of error is 140, is it normal? I was expecting much less than that</strong>)</li>
  <li>After the fourth iteration the costs of the error start increasing (<strong>I don't know if it is normal or not?</strong>)</li>
  </ul>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-01-26T13:02:48.307Z
Comments.Text: I would always output the average error, not just the average cost. For the average error of an approximation of sine 140 is way too high, obviously. I would also restrict the input to a certain range. It might be the case that the NN doesn't generalise to bigger inputs. Output your error for the training and the test set. And finally: Try a learning rate so small that almost nothing happens, then slowly increase. It might be that you are jumping around too much in the solution space.
Comments.UserDisplayName: BlindKungFuMaster
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-01-18T05:17:52.317Z
Comments.Text: As you progress, you might want to try Residual connections to help with vanishing gradients. Search ResNets. Andrew Ng's Deep Learning ResNet video should be on Youtube. Check it out!
Comments.UserDisplayName: Ali_Ayub
Tags: <neural-networks><backpropagation>
Title: How to test if my implementation of back propagation neural Network is correct
UserDisplayName: Anas EL KORCHI
ThreadText: # How to test if my implementation of back propagation neural Network is correct
  
  (Asked by: username_0 on 2017-01-26)
  
  I am working on an implementation of the back propagation algorithm. What I have implemented so far seems working but I can't be sure that the algorithm is well implemented, here is what I have noticed during training test of my network:
  
  Specification of the implementation:
  
  * A data set containing almost 100000 raw containing (3 variable as input, the sinus of the sum of those three variables as expected output).
  * The network does have 7 layers, all the layers use the sigmoid activation function
  
  When I run the back propagation training process:
  
  * The minimum of costs of the error is found at the fourth iteration (**The minimum cost of error is 140, is it normal? I was expecting much less than that**)
  * After the fourth iteration the costs of the error start increasing (**I don't know if it is normal or not?**)
  
  **Tags:** neural-networks, backpropagation
  > **Comments:**
  > **username_1**: I would always output the average error, not just the average cost. For the average error of an approximation of sine 140 is way too high, obviously. I would also restrict the input to a certain range. It might be the case that the NN doesn't generalise to bigger inputs. Output your error for the training and the test set. And finally: Try a learning rate so small that almost nothing happens, then slowly increase. It might be that you are jumping around too much in the solution space.  
  > **username_2**: As you progress, you might want to try Residual connections to help with vanishing gradients. Search ResNets. Andrew Ng's Deep Learning ResNet video should be on Youtube. Check it out!  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_0 on 2017-01-27. Score: 4 | Accepted answer
  
  Actually the implementation was correct,
  
  The source of the problem that causes a big error and really slow learning was the architecture of the neural network it self, the ANN has 7 hidden layers which causes the vanishing gradient problem.
  
  When I have decreased the ANN layers to 3 the cost of error was widely reduced besides of that the learning process was faster.
  
  Another common solution is to use RELU or ELU or SELU in the hidden layers of the neural network instead of the sigmoid function
  
  > **Comments:**
  > **username_0**: if you change the activation function to relu or elu that would fix the issue  
  > **username_5**: Yet another common solution is residual network, i.e., adding fixed weight connections skipping layers. I like it most as it works without modifying the structure / activation function.  
  
  ---
  
  ### Answer by username_3 on 2018-02-28. Score: 3
  
  I believe the best way to do this is using **numerical gradient**. To understand the concept, we need to look the definition of derivatives using limits:
  
  [![enter image description here](https://i.sstatic.net/9Qanu.png)](https://i.sstatic.net/9Qanu.png)
  
  It means that, when you don't know how to derive some formula (or you just don't want to), you can approximate it by computing the output for a small change in input, subtract from the original result (no change), and normalize by this change.
  
  Example:
  We know the derivative of f(x) = x^2 is f'(x) = 2x. But, let suppose we don't and we are using x = 3 and h = 0.001 (it tends to zero in fact):
  
  f(3 + 0.001) = (3 + 0.001)^2 = 9,006 (approximately)   
  f(3) = 3^2 = 9
  
  Thus,
  
  (9,006 - 9) / 0.001 = 6
  
  It's is exactly to f'(3) = 2\*3 = 6.
  
  In practice, if you want to know if your backpropagation is correct,
  
  1. pass a single example (x1) through your network and computes the output (o1). Compute the gradient with respect to x1 (d1).
  2. Then, add a small value to the input (h), pass through the network again, and computes the new output (o2).
  3. Subtract o2 - o1 and divide by h. It must be close enough to d1.
  
  That's it. It's called **gradient checking**. I hope it helps.
  
  
  ---
  
  ### Answer by username_4 on 2017-06-28. Score: 2
  
  One thing that you should know is that the sigmoid function limits the output to a value between 0 and 1, which means that using a lot of hidden layers will lead quickly to a Vanishing Gradient.
  
  Try to use relu activation function, it has the property to output all the information it gets from the previous layer.
  
  
  ---
  
  