Answers.Body: <p>The given so far answers focus on numerical methods to check your gradients. It is <em>really</em> useful, especially if one doesn't have much experience in backprop.</p>
  
  <p>But I'd like to add here a pure practical "sanity check", relatively fast and easy to perform, which also works for other issues, e.g. (rough) hyperparameters selection. To see if your network makes sense, reduce the training set to a few examples and try to overfit the network. If the loss falls to zero and training accuracy skyrockets to 1, it means that both passes work correctly and can move on to the real training. Otherwise, something's not right and should dive into specific parts of the network, in particular check the gradients numerically.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-10-07T23:40:18.953Z
Answers.LastActivityDate: 2017-10-07T23:40:18.953Z
Answers.UserDisplayName: Maxim
Answers.Body: <p>For future reference, you can check your correctness by the finite difference method.</p>
  
  <p><a href="http://www.cedar.buffalo.edu/~srihari/CSE574/Chap5/Chap5.3-BackProp.pdf" rel="nofollow noreferrer">http://www.cedar.buffalo.edu/~srihari/CSE574/Chap5/Chap5.3-BackProp.pdf</a> (p.23)</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-09-03T14:29:09.647Z
Answers.LastActivityDate: 2017-09-03T14:29:09.647Z
Answers.UserDisplayName: k.c. sayz 'k.c sayz'
Answers.Body: <p>Don't feel too bad for having gotten it slightly wrong because backpropagation is notoriously difficult to implement <a href="https://web.archive.org/web/20171122205139/http://ufldl.stanford.edu/wiki/index.php/Gradient_checking_and_advanced_optimization" rel="nofollow noreferrer">[1]</a>.</p>
  <p>There is a technique called <a href="https://web.archive.org/web/20171122205139/http://ufldl.stanford.edu/wiki/index.php/Gradient_checking_and_advanced_optimization" rel="nofollow noreferrer"><strong>gradient checking</strong></a>, which you can implement to test the correctness of your backpropagation implementation. I would argue that even gradient checking is a little tricky to implement.</p>
  <p>How does gradient checking work?</p>
  <ul>
  <li><p>Backpropagation computes the gradients <span class="math-container">$\frac{\partial J}{\partial \theta}$</span>, where <span class="math-container">$\theta$</span> denotes the parameters of the model. <span class="math-container">$J$</span> is computed using forward propagation and your loss function.</p>
  </li>
  <li><p>But because forward propagation is fairly straightforward to implement, most people are usually confident that you got its implementation correct. So, the trick is to use the value of <span class="math-container">$J$</span> to verify your code for computing <span class="math-container">$\frac{\partial J}{\partial \theta}$</span>.</p>
  </li>
  <li><p>We know that by definition, the gradient or derivative is given by:
  <span class="math-container">$$\frac{\partial J}{\partial \theta} = \lim_{\epsilon \rightarrow 0} \frac{J(\theta + \epsilon) − J(\theta − \epsilon)}{2 \epsilon}$$</span>
  Since we trust our calculation of <span class="math-container">$J$</span>, we can easily compute the value of <span class="math-container">$J(\theta + \epsilon) − J(\theta − \epsilon)$</span>.</p>
  </li>
  </ul>
  <p>For more information, see <a href="https://www.coursera.org/learn/machine-learning/lecture/Y3s6r/gradient-checking" rel="nofollow noreferrer">this Andrew Ng's video lecture</a> and <a href="https://web.archive.org/web/20171122205139/http://ufldl.stanford.edu/wiki/index.php/Gradient_checking_and_advanced_optimization" rel="nofollow noreferrer">these notes</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-09-09T14:34:41.440Z
Answers.LastActivityDate: 2021-12-11T10:20:08.447Z
Answers.LastEditDate: 2021-12-11T10:20:08.447Z
Answers.UserDisplayName: Tshilidzi Mudau
Body: <p>I'm working on an implementation of the backpropagation algorithm for a simple neural network, which predicts a probability of survival (1 or 0).</p>
  <p>However, I can't get it above 80%, no matter how much I try to set the right hyperparameters. I suspect that's because my backpropagation is implemented incorrectly, since I tried 2 different types of code and both give me the same results.</p>
  <p>Is there a way to determine whether <a href="https://pastebin.com/4PmYFAjH" rel="nofollow noreferrer">my implementation</a> of backpropagation is correct?</p>
  
Tags: <neural-networks><backpropagation><algorithm-request>
Title: How do I know if my backpropagation is implemented correctly?
UserDisplayName: Damian Matkowski
ThreadText: # How do I know if my backpropagation is implemented correctly?
  
  (Asked by: username_0 on 2017-09-03)
  
  I'm working on an implementation of the backpropagation algorithm for a simple neural network, which predicts a probability of survival (1 or 0).
  
  However, I can't get it above 80%, no matter how much I try to set the right hyperparameters. I suspect that's because my backpropagation is implemented incorrectly, since I tried 2 different types of code and both give me the same results.
  
  Is there a way to determine whether [my implementation](https://pastebin.com/4PmYFAjH) of backpropagation is correct?
  
  **Tags:** neural-networks, backpropagation, algorithm-request
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2017-09-09. Score: 4 | Accepted answer
  
  Don't feel too bad for having gotten it slightly wrong because backpropagation is notoriously difficult to implement [[1]](https://web.archive.org/web/20171122205139/http://ufldl.stanford.edu/wiki/index.php/Gradient_checking_and_advanced_optimization).
  
  There is a technique called [**gradient checking**](https://web.archive.org/web/20171122205139/http://ufldl.stanford.edu/wiki/index.php/Gradient_checking_and_advanced_optimization), which you can implement to test the correctness of your backpropagation implementation. I would argue that even gradient checking is a little tricky to implement.
  
  How does gradient checking work?
  
  * Backpropagation computes the gradients $\frac{\partial J}{\partial \theta}$, where $\theta$ denotes the parameters of the model. $J$ is computed using forward propagation and your loss function.
  * But because forward propagation is fairly straightforward to implement, most people are usually confident that you got its implementation correct. So, the trick is to use the value of $J$ to verify your code for computing $\frac{\partial J}{\partial \theta}$.
  * We know that by definition, the gradient or derivative is given by:
    $$\frac{\partial J}{\partial \theta} = \lim\_{\epsilon \rightarrow 0} \frac{J(\theta + \epsilon) − J(\theta − \epsilon)}{2 \epsilon}$$
    Since we trust our calculation of $J$, we can easily compute the value of $J(\theta + \epsilon) − J(\theta − \epsilon)$.
  
  For more information, see [this Andrew Ng's video lecture](https://www.coursera.org/learn/machine-learning/lecture/Y3s6r/gradient-checking) and [these notes](https://web.archive.org/web/20171122205139/http://ufldl.stanford.edu/wiki/index.php/Gradient_checking_and_advanced_optimization).
  
  
  ---
  
  ### Answer by username_1 on 2017-10-07. Score: 1
  
  The given so far answers focus on numerical methods to check your gradients. It is *really* useful, especially if one doesn't have much experience in backprop.
  
  But I'd like to add here a pure practical "sanity check", relatively fast and easy to perform, which also works for other issues, e.g. (rough) hyperparameters selection. To see if your network makes sense, reduce the training set to a few examples and try to overfit the network. If the loss falls to zero and training accuracy skyrockets to 1, it means that both passes work correctly and can move on to the real training. Otherwise, something's not right and should dive into specific parts of the network, in particular check the gradients numerically.
  
  
  ---
  
  ### Answer by username_2 on 2017-09-03. Score: 1
  
  For future reference, you can check your correctness by the finite difference method.
  
  <http://www.cedar.buffalo.edu/~srihari/CSE574/Chap5/Chap5.3-BackProp.pdf> (p.23)
  
  
  ---
  
  