Answers.Body: <p>There may be a confusion between <em>root finding</em> and <em>optimization</em>.</p>
  <p>Your formula appears designed to find a point where a scalar function <span class="math-container">$f$</span> of several variables is equal to zero. However, this goal is not particularly useful. Generically, a function of <span class="math-container">$n$</span> variables equals zero on some <span class="math-container">$(n - 1)$</span>-dimensional space. Your formula may converge to some point in this space. But <em>why</em> does one want to find such a point?</p>
  <p>If the goal is fitting a model, and <span class="math-container">$f$</span> is a loss function, one wants to find a point where <span class="math-container">$f$</span> is <em>minimized</em>, not where <span class="math-container">$f = 0$</span>. (Note, even if somehow one knew a priori that <span class="math-container">$f$</span>'s minimum value is zero, Newton's method would be an <a href="https://math.stackexchange.com/q/857250/820184">inefficient</a> way to find it.)</p>
  <p>One can also frame minimization as finding a point where <span class="math-container">$\nabla f = 0$</span>, but this is finding a root of a <em>vector</em> function (i.e., simultaneous root of <span class="math-container">$n$</span> scalar functions), not of a single scalar function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-01T21:03:12.220Z
Answers.LastActivityDate: 2024-08-01T21:03:12.220Z
Answers.UserDisplayName: nanoman
Answers.Body: <p><a href="https://en.wikipedia.org/wiki/Automatic_differentiation" rel="nofollow noreferrer">Automatic Differentiation (AD)</a> is a technique used to compute the derivatives of functions with respect to their inputs applying chain rules, and the famous <a href="https://en.wikipedia.org/wiki/Backpropagation" rel="nofollow noreferrer">backpropagation</a> algo used in most ANNs is a specific reverse mode of AD. They're highly effective for calculating first order gradients together with SGD empirically in ML and other fields and are used by many frameworks like TensorFlow, PyTorch and JAX.</p>
  <p>On the other hand, your referenced method involves essentially a nonlinearly scaled version of the gradient which can lead to instability or divergence in certain cases especially if the gradient value in the <em>denominator</em> is very small or zero. And if it's very large you update becomes also very slow. In practice algorithm stability and robustness are critical in ML where techniques like SGD with adaptive learning rates (e.g., Adam, RMSprop) have been designed well to handle issues of step size adaptation and convergence. Therefore in ML your proposed 'scaled' gradient algorithm is not used compared to the vanilla gradient AD algorithms due to its instability and extra problematic inverse computation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-01T08:38:26.987Z
Answers.LastActivityDate: 2024-08-01T09:04:41.833Z
Answers.LastEditDate: 2024-08-01T09:04:41.833Z
Answers.UserDisplayName: cinch
Body: <p><span class="math-container">$$ w_{n+1} = w_n - \alpha \frac{f \nabla f}{||\nabla f||^2} $$</span>
  It calculates where the line that’s in the same direction as gradient and has <span class="math-container">$f$</span>’s slope in that direction becomes zero.
  In one dimensional case it becomes:
  <span class="math-container">$$ [x_{n+1}] = [x_n] - \alpha f(x_n) \frac{[f’(x_n)]}{||[f’(x_n)]||^2}$$</span></p>
  <p><span class="math-container">$$x_{n+1} = x_n - \alpha \frac{f(x_n)}{f’(x_n)}$$</span></p>
  <p>Which is just Newton’s root finding method if <span class="math-container">$\alpha = 1$</span>.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-08-01T14:07:31.600Z
Comments.Text: Used where? Can you be more specific? Right now, this could not even be related to AI.
Comments.UserDisplayName: nbro
Tags: <optimization>
Title: Why isn’t this formula used at all?
UserDisplayName: hansoko
ThreadText: # Why isn’t this formula used at all?
  
  (Asked by: username_0 on 2024-08-01)
  
  $$ w\_{n+1} = w\_n - \alpha \frac{f \nabla f}{||\nabla f||^2} $$
  It calculates where the line that’s in the same direction as gradient and has $f$’s slope in that direction becomes zero.
  In one dimensional case it becomes:
  $$ [x\_{n+1}] = [x\_n] - \alpha f(x\_n) \frac{[f’(x\_n)]}{||[f’(x\_n)]||^2}$$
  
  $$x\_{n+1} = x\_n - \alpha \frac{f(x\_n)}{f’(x\_n)}$$
  
  Which is just Newton’s root finding method if $\alpha = 1$.
  
  **Tags:** optimization
  
  > **Comments:**
  > **username_1**: Used where? Can you be more specific? Right now, this could not even be related to AI.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2024-08-01. Score: 4 | Accepted answer
  
  [Automatic Differentiation (AD)](https://en.wikipedia.org/wiki/Automatic_differentiation) is a technique used to compute the derivatives of functions with respect to their inputs applying chain rules, and the famous [backpropagation](https://en.wikipedia.org/wiki/Backpropagation) algo used in most ANNs is a specific reverse mode of AD. They're highly effective for calculating first order gradients together with SGD empirically in ML and other fields and are used by many frameworks like TensorFlow, PyTorch and JAX.
  
  On the other hand, your referenced method involves essentially a nonlinearly scaled version of the gradient which can lead to instability or divergence in certain cases especially if the gradient value in the *denominator* is very small or zero. And if it's very large you update becomes also very slow. In practice algorithm stability and robustness are critical in ML where techniques like SGD with adaptive learning rates (e.g., Adam, RMSprop) have been designed well to handle issues of step size adaptation and convergence. Therefore in ML your proposed 'scaled' gradient algorithm is not used compared to the vanilla gradient AD algorithms due to its instability and extra problematic inverse computation.
  
  
  ---
  
  ### Answer by username_2 on 2024-08-01. Score: 1
  
  There may be a confusion between *root finding* and *optimization*.
  
  Your formula appears designed to find a point where a scalar function $f$ of several variables is equal to zero. However, this goal is not particularly useful. Generically, a function of $n$ variables equals zero on some $(n - 1)$-dimensional space. Your formula may converge to some point in this space. But *why* does one want to find such a point?
  
  If the goal is fitting a model, and $f$ is a loss function, one wants to find a point where $f$ is *minimized*, not where $f = 0$. (Note, even if somehow one knew a priori that $f$'s minimum value is zero, Newton's method would be an [inefficient](https://math.stackexchange.com/q/857250/820184) way to find it.)
  
  One can also frame minimization as finding a point where $\nabla f = 0$, but this is finding a root of a *vector* function (i.e., simultaneous root of $n$ scalar functions), not of a single scalar function.
  
  
  ---
  
  