Answers.Body: <p>This link from Stanford is by far the best resource on gradient checking that I have encountered so far:</p>
  
  <p><a href="http://cs231n.github.io/neural-networks-3/" rel="nofollow noreferrer">http://cs231n.github.io/neural-networks-3/</a></p>
  
  <p><strong>I am sure</strong> it will help you a lot.</p>
  
  <p><strong>Pro Tip</strong>: make sure that you use <strong>"centered/central difference"</strong> formula for the derivative calculations and also use <strong>"relative error"</strong> (not absolute) to compare the two gradients.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-28T13:58:26.100Z
Answers.LastActivityDate: 2018-05-28T13:58:26.100Z
Answers.UserDisplayName: pcko1
Answers.Body: <p>You should be able to do gradient checking as long as you fix the randomness by fixing the random seed, on python you might want to look at numpy.random.seed.</p>
  
  <p>From <a href="http://cs231n.github.io/neural-networks-3/#ensemble" rel="nofollow noreferrer">http://cs231n.github.io/neural-networks-3/#ensemble</a> :</p>
  
  <blockquote>
    <p>When performing gradient check, remember to turn off any non-deterministic effects in the network, such as dropout, random data augmentations, etc. Otherwise these can clearly introduce huge errors when estimating the numerical gradient. The downside of turning off these effects is that you wouldn’t be gradient checking them (e.g. it might be that dropout isn’t backpropagated correctly). <strong>Therefore, a better solution might be to force a particular random seed before evaluating both (f(x+h)) and (f(x-h)), and when evaluating the analytic gradient.</strong></p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-25T00:54:08.627Z
Answers.LastActivityDate: 2018-04-25T00:54:08.627Z
Answers.UserDisplayName: k.c. sayz 'k.c sayz'
Body: <p>I have implemented a neural network (NN) using python and numpy only for learning purposes. I have already coded learning rate, momentum, and L1/L2 regularization and checked the implementation with gradient checking. </p>
  
  <p>A few days ago, I implemented batch normalization using the formulas provided by the <a href="https://arxiv.org/pdf/1502.03167.pdf" rel="nofollow noreferrer">original paper</a>. However, in contrast with learning/momentum/regularization, the batch normalization procedure behaves differently during <strong>fit</strong> and <strong>predict</strong> phases - both needed for gradient checking. As we fit the network, batch normalization computes each batch mean and estimates the population's mean to be used when we want to predict something.</p>
  
  <p>In a similar way, I know we may not perform gradient checking in a neural network with <strong>dropout</strong>, since dropout turns some gradients to zero during <em>fit</em> and is not applied during <em>prediction</em>.</p>
  
  <p>Can we perform gradient checking in NN with batch normalization? If so, how?</p>
  
Tags: <neural-networks><python><gradient-descent><batch-normalization>
Title: How to perform gradient checking in a neural network with batch normalization?
UserDisplayName: Arnaldo Gualberto
ThreadText: # How to perform gradient checking in a neural network with batch normalization?
  
  (Asked by: username_0 on 2018-04-24)
  
  I have implemented a neural network (NN) using python and numpy only for learning purposes. I have already coded learning rate, momentum, and L1/L2 regularization and checked the implementation with gradient checking.
  
  A few days ago, I implemented batch normalization using the formulas provided by the [original paper](https://arxiv.org/pdf/1502.03167.pdf). However, in contrast with learning/momentum/regularization, the batch normalization procedure behaves differently during **fit** and **predict** phases - both needed for gradient checking. As we fit the network, batch normalization computes each batch mean and estimates the population's mean to be used when we want to predict something.
  
  In a similar way, I know we may not perform gradient checking in a neural network with **dropout**, since dropout turns some gradients to zero during *fit* and is not applied during *prediction*.
  
  Can we perform gradient checking in NN with batch normalization? If so, how?
  
  **Tags:** neural-networks, python, gradient-descent, batch-normalization
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2018-05-28. Score: 0
  
  This link from Stanford is by far the best resource on gradient checking that I have encountered so far:
  
  <http://cs231n.github.io/neural-networks-3/>
  
  **I am sure** it will help you a lot.
  
  **Pro Tip**: make sure that you use **"centered/central difference"** formula for the derivative calculations and also use **"relative error"** (not absolute) to compare the two gradients.
  
  
  ---
  
  ### Answer by username_2 on 2018-04-25. Score: 0
  
  You should be able to do gradient checking as long as you fix the randomness by fixing the random seed, on python you might want to look at numpy.random.seed.
  
  From <http://cs231n.github.io/neural-networks-3/#ensemble> :
  
  > When performing gradient check, remember to turn off any non-deterministic effects in the network, such as dropout, random data augmentations, etc. Otherwise these can clearly introduce huge errors when estimating the numerical gradient. The downside of turning off these effects is that you wouldn’t be gradient checking them (e.g. it might be that dropout isn’t backpropagated correctly). **Therefore, a better solution might be to force a particular random seed before evaluating both (f(x+h)) and (f(x-h)), and when evaluating the analytic gradient.**
  
  
  ---
  
  