Answers.Body: <p>Nope.  Delta rule: taylor series with truncation relates mean, variance, and gradient.</p>
  <p>Variance is proportional to the square of the local slope in the region. If your minimum is at the bottom of a wide-area (think grand canyon) then you can have low variance.  If the minimum is at the bottom of a well, then the local variance can be high.</p>
  <p>The goal of optimizers (like those with momentum and adaptivity) is to test the local area and maybe bump the learner out of a local but not global best to something better nearby.  That means you aren't exactly testing the point as much as you are testing the space around the point.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-07T13:16:32.487Z
Answers.LastActivityDate: 2022-06-07T13:16:32.487Z
Answers.UserDisplayName: EngrStudent
Body: <p>As I understand the gradient should reflect how near the weights are to the optimal values. In this way i will expect that on the first epochs the gradients far from zero or at least not mostly zero and as we train the net the gradients will arrive to values nearest to zero.
  But it is not the case as you can see for example here (This image show gradients distribution on each epoch):</p>
  <p><a href="https://i.sstatic.net/YG5IN.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/YG5IN.png" alt="enter image description here" /></a></p>
  <p><a href="https://wandb.ai/ayush-thakur/debug-neural-nets/reports/Visualizing-and-Debugging-Neural-Networks-with-PyTorch-and-W-B--Vmlldzo2OTUzNA" rel="nofollow noreferrer">https://wandb.ai/ayush-thakur/debug-neural-nets/reports/Visualizing-and-Debugging-Neural-Networks-with-PyTorch-and-W-B--Vmlldzo2OTUzNA</a></p>
  <p>and here (This image show gradients for 5 layers after the first batch):</p>
  <p><a href="https://i.sstatic.net/Ad1Hs.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Ad1Hs.png" alt="enter image description here" /></a></p>
  <p><a href="https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/tutorial3/Activation_Functions.html" rel="nofollow noreferrer">https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/tutorial3/Activation_Functions.html</a></p>
  <p>I've seen the same behavior in other simple nets.
  Can someone explain this?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-09T20:14:35.103Z
Comments.Text: I'm not sure how meaningful gradient distributions are. Try observing L2 norm of gradients. That should get smaller as your model converges.
Comments.UserDisplayName: SpiderRico
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-09T21:51:45.310Z
Comments.Text: Thanks for your response. I've tried but the distribution is basically the same (centered on zero). Why do you think that L2 is better?
Comments.UserDisplayName: Guillermo Alvarez
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-09T22:00:13.700Z
Comments.Text: L2 norm of the gradient kinda tells you, on aggregate, whether your gradient gets smaller or not. Individual dimensions may vary. L2 norm of a vector is just a scalar value, you can just observe how this changes over the training,
Comments.UserDisplayName: SpiderRico
Tags: <deep-learning><training><gradient-descent><gradient>
Title: How to explain near zero gradients on first epochs?
UserDisplayName: Guillermo Alvarez
ThreadText: # How to explain near zero gradients on first epochs?
  
  (Asked by: username_0 on 2022-05-09)
  
  As I understand the gradient should reflect how near the weights are to the optimal values. In this way i will expect that on the first epochs the gradients far from zero or at least not mostly zero and as we train the net the gradients will arrive to values nearest to zero.
  But it is not the case as you can see for example here (This image show gradients distribution on each epoch):
  
  [![enter image description here](https://i.sstatic.net/YG5IN.png)](https://i.sstatic.net/YG5IN.png)
  
  <https://wandb.ai/ayush-thakur/debug-neural-nets/reports/Visualizing-and-Debugging-Neural-Networks-with-PyTorch-and-W-B--Vmlldzo2OTUzNA>
  
  and here (This image show gradients for 5 layers after the first batch):
  
  [![enter image description here](https://i.sstatic.net/Ad1Hs.png)](https://i.sstatic.net/Ad1Hs.png)
  
  <https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/tutorial3/Activation_Functions.html>
  
  I've seen the same behavior in other simple nets.
  Can someone explain this?
  
  **Tags:** deep-learning, training, gradient-descent, gradient
  
  > **Comments:**
  > **username_1**: I'm not sure how meaningful gradient distributions are. Try observing L2 norm of gradients. That should get smaller as your model converges.  
  > **username_0**: Thanks for your response. I've tried but the distribution is basically the same (centered on zero). Why do you think that L2 is better?  
  > **username_1**: L2 norm of the gradient kinda tells you, on aggregate, whether your gradient gets smaller or not. Individual dimensions may vary. L2 norm of a vector is just a scalar value, you can just observe how this changes over the training,  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-06-07. Score: 0
  
  Nope. Delta rule: taylor series with truncation relates mean, variance, and gradient.
  
  Variance is proportional to the square of the local slope in the region. If your minimum is at the bottom of a wide-area (think grand canyon) then you can have low variance. If the minimum is at the bottom of a well, then the local variance can be high.
  
  The goal of optimizers (like those with momentum and adaptivity) is to test the local area and maybe bump the learner out of a local but not global best to something better nearby. That means you aren't exactly testing the point as much as you are testing the space around the point.
  
  
  ---
  
  