Answers.Body: <p>Momentum was big.  It allowed several steps to be evened out so that most of the motion in the weights was in the direction of the optimum.  It operates against sequential measurements of the error.  This means that several estimates of the gradient give better local picture of the loss-surface.</p>
  <p>The error has a magnitude, so the gradient has both direction and magnitude.  It tells us the direction to go, but also how far to go.  The space is complex, so many momentum methods smooth the magnitude and direction by combining thousands of gradient estimates.</p>
  <p>Dataset distillation is interesting because it can require a 10x larger network to learn a task, then distill that learning into a 1x network to do the task.  This is a universal-to-specific transformation, using a global approximator to find the local landscape that works is very different than building an approximator for use only within that landscape.  The converged space of both networks is the same, but one is contrived in vastly fewer parameters.  The gradient in the around the optimum, the perturbed gradient given the training data, tells how to perform the simplification.</p>
  <p><a href="https://arxiv.org/pdf/1312.6034.pdf" rel="nofollow noreferrer">Saliency maps</a> use back-propagation (single-pass gradient on fully trained networks) to infer interior structure and operations of complex neural networks.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-28T15:17:54.480Z
Answers.LastActivityDate: 2022-11-28T15:40:50.057Z
Answers.LastEditDate: 2022-11-28T15:40:50.057Z
Answers.UserDisplayName: EngrStudent
Body: <p>Gradients are used in optimization algorithms.</p>
  <p>I know that a gradient gives us information about the direction in which one needs to update the weights of a neural network. We need to travel in the opposite direction of the gradient to get optimal values.</p>
  <p>Thus the gradient provides <strong>direction to update parameters</strong>.</p>
  <p>Is it the only information provided by the gradient? Or does it provide any other information that helps in the training process?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-19T03:18:46.683Z
Comments.Text: gradient magnitudes also tell you how much you need to update the weights, in addition to the direction (i.e., increment or decrement).
Comments.UserDisplayName: SpiderRico
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-19T09:02:38.303Z
Comments.Text: @SpiderRico If it is contributing in terms of magnitude, I feel the learning rate is decreasing the magnitude... So, I'm feeling that the magnitude of the gradient is just nominal.. Is it true?
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-19T12:09:34.070Z
Comments.Text: Yeah you can adjust for that with learning rate. However, note that magnitude of a gradient is not the same for every dimension. So, some dimensions would still be changed more than the others.
Comments.UserDisplayName: SpiderRico
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-19T12:56:16.583Z
Comments.Text: @SpiderRico Feel free to provide a formal answer if you think you have enough information to at least partially answer the question.
Comments.UserDisplayName: nbro
Tags: <optimization><gradient><calculus>
Title: What all does the gradient tells us other than the direction to move parameters?
UserDisplayName: hanugm
ThreadText: # What all does the gradient tells us other than the direction to move parameters?
  
  (Asked by: username_0 on 2021-08-19)
  
  Gradients are used in optimization algorithms.
  
  I know that a gradient gives us information about the direction in which one needs to update the weights of a neural network. We need to travel in the opposite direction of the gradient to get optimal values.
  
  Thus the gradient provides **direction to update parameters**.
  
  Is it the only information provided by the gradient? Or does it provide any other information that helps in the training process?
  
  **Tags:** optimization, gradient, calculus
  > **Comments:**
  > **username_1**: gradient magnitudes also tell you how much you need to update the weights, in addition to the direction (i.e., increment or decrement).  
  > **username_0**: @SpiderRico If it is contributing in terms of magnitude, I feel the learning rate is decreasing the magnitude... So, I'm feeling that the magnitude of the gradient is just nominal.. Is it true?  
  > **username_1**: Yeah you can adjust for that with learning rate. However, note that magnitude of a gradient is not the same for every dimension. So, some dimensions would still be changed more than the others.  
  > **username_2**: @SpiderRico Feel free to provide a formal answer if you think you have enough information to at least partially answer the question.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2022-11-28. Score: 1
  
  Momentum was big. It allowed several steps to be evened out so that most of the motion in the weights was in the direction of the optimum. It operates against sequential measurements of the error. This means that several estimates of the gradient give better local picture of the loss-surface.
  
  The error has a magnitude, so the gradient has both direction and magnitude. It tells us the direction to go, but also how far to go. The space is complex, so many momentum methods smooth the magnitude and direction by combining thousands of gradient estimates.
  
  Dataset distillation is interesting because it can require a 10x larger network to learn a task, then distill that learning into a 1x network to do the task. This is a universal-to-specific transformation, using a global approximator to find the local landscape that works is very different than building an approximator for use only within that landscape. The converged space of both networks is the same, but one is contrived in vastly fewer parameters. The gradient in the around the optimum, the perturbed gradient given the training data, tells how to perform the simplification.
  
  [Saliency maps](https://arxiv.org/pdf/1312.6034.pdf) use back-propagation (single-pass gradient on fully trained networks) to infer interior structure and operations of complex neural networks.
  
  
  ---
  
  