Body: <p>I am troubled by natural gradient methods.
  If we have a function f(x) we wish to minimize, gradient descent minimizes f(x) of course, but what does the natural gradient do?</p>
  <p>I found on <a href="https://towardsdatascience.com/natural-gradient-ce454b3dcdfa" rel="nofollow noreferrer">https://towardsdatascience.com/natural-gradient-ce454b3dcdfa</a>:</p>
  <blockquote>
  <p>Instead of fixing the euclidean distance each parameter moves(distance in the parameter space), we can fix the distance in the distribution space of the target output.</p>
  </blockquote>
  <p>Where did the distributions come from? If we wish to minimize f(x), the target output is just a minimizer x* right, and not a distribution, or am I missing something?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-31T12:16:46.620Z
Comments.Text: You're asking multiple distinct questions here. The question in the title is quite general and different from the question "Where did the distributions come from?". So, please, edit your post to leave **only one question**.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-31T17:35:45.157Z
Comments.Text: https://wiseodd.github.io/techblog/2018/03/14/natural-gradient/#:~:text=Up%20to%20constant%20factor%20of,%E2%88%87%CE%B8L(%CE%B8).
Comments.UserDisplayName: Abhishek Verma
Tags: <deep-learning><gradient-descent><geometric-deep-learning><information-theory>
Title: Do Gradient Descent and Natural Gradient solve the same problem?
UserDisplayName: Galois
ThreadText: # Do Gradient Descent and Natural Gradient solve the same problem?
  
  (Asked by: username_0 on 2021-03-31)
  
  I am troubled by natural gradient methods.
  If we have a function f(x) we wish to minimize, gradient descent minimizes f(x) of course, but what does the natural gradient do?
  
  I found on <https://towardsdatascience.com/natural-gradient-ce454b3dcdfa>:
  
  > Instead of fixing the euclidean distance each parameter moves(distance in the parameter space), we can fix the distance in the distribution space of the target output.
  
  Where did the distributions come from? If we wish to minimize f(x), the target output is just a minimizer x\* right, and not a distribution, or am I missing something?
  
  **Tags:** deep-learning, gradient-descent, geometric-deep-learning, information-theory
  > **Comments:**
  > **username_1**: You're asking multiple distinct questions here. The question in the title is quite general and different from the question "Where did the distributions come from?". So, please, edit your post to leave **only one question**.  
  > **username_2**: https://wiseodd.github.io/techblog/2018/03/14/natural-gradient/#:~:text=Up%20to%20constant%20factor%20of,%E2%88%87%CE%B8L(%CE%B8).  
  
  ---
  
  ## 0 answers
  
  