Answers.Body: <p>Your attempt to find an optimal adaptive learning rate at each update step makes sense, however, even the finite difference Newton-Ralphson application for your one-dimensional problem is likely sensitive and unstable numerically since the loss landscape in deep learning is nonconvex (far from quadratic as the classic Newton-Ralphson assumes) even locally. This can lead to learning rate updates unstable. You might try out more stable quasi-Newton methods such as <a href="https://en.wikipedia.org/wiki/Broyden%E2%80%93Fletcher%E2%80%93Goldfarb%E2%80%93Shanno_algorithm" rel="nofollow noreferrer">BFGS</a>, but they share the same sensitivity issue for second-order derivatives.</p>
  <p>Also if you combine your method with mini-batch like most other deep learning stochastic optimization methods, they're incompatible since the deterministic result of Newton-Ralphson in one mini-batch may not generalize well to the next batch, leading to inconsistent update.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-04T19:45:24.193Z
Answers.LastActivityDate: 2024-11-04T19:45:24.193Z
Answers.UserDisplayName: cinch
Body: <p>My idea is simple, for a loss function <span class="math-container">$L$</span>, learning rate <span class="math-container">$\alpha$</span> and weights <span class="math-container">$W_n$</span> define the function <span class="math-container">$u_n(\alpha)$</span> as:
  <span class="math-container">$$u_n(\alpha) = L(W_n - \alpha \nabla_n L)$$</span>
  If we find the minimum of <span class="math-container">$u_n$</span>, we’ve found the best learning rate for <span class="math-container">$\nabla_n L$</span>. Using Newton-Raphson method to approximate the point in which <span class="math-container">$u_n’$</span> is zero we can find the minimum of <span class="math-container">$u_n$</span>:
  <span class="math-container">$$\alpha_{n+1} = \alpha_n - \frac{u_n’}{u_n’’}$$</span>
  Finally update the weights:
  <span class="math-container">$$W_{n+1} = W_n - \alpha_{n+1} \nabla L$$</span></p>
  <p>I used this method to train a neural network on MNIST(calculated <span class="math-container">$u_n’$</span> and <span class="math-container">$u_n’’$</span> using the finite difference method), and while it was better than regular SGD, it wasn’t impressive compared to Adam. Why doesn’t this method work well? Is there any way to improve it?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-11-04T11:59:35.320Z
Comments.Text: Could you post some learning curves comparing your method vs SGD vs Adam? I think it could help to figure out an answer
Comments.UserDisplayName: Luca Anzalone
Tags: <neural-networks><optimization>
Title: Why doesn’t this optimization method work?
UserDisplayName: hansoko
ThreadText: # Why doesn’t this optimization method work?
  
  (Asked by: username_0 on 2024-11-04)
  
  My idea is simple, for a loss function $L$, learning rate $\alpha$ and weights $W\_n$ define the function $u\_n(\alpha)$ as:
  $$u\_n(\alpha) = L(W\_n - \alpha \nabla\_n L)$$
  If we find the minimum of $u\_n$, we’ve found the best learning rate for $\nabla\_n L$. Using Newton-Raphson method to approximate the point in which $u\_n’$ is zero we can find the minimum of $u\_n$:
  $$\alpha\_{n+1} = \alpha\_n - \frac{u\_n’}{u\_n’’}$$
  Finally update the weights:
  $$W\_{n+1} = W\_n - \alpha\_{n+1} \nabla L$$
  
  I used this method to train a neural network on MNIST(calculated $u\_n’$ and $u\_n’’$ using the finite difference method), and while it was better than regular SGD, it wasn’t impressive compared to Adam. Why doesn’t this method work well? Is there any way to improve it?
  
  **Tags:** neural-networks, optimization
  > **Comments:**
  > **username_1**: Could you post some learning curves comparing your method vs SGD vs Adam? I think it could help to figure out an answer  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-11-04. Score: 1 | Accepted answer
  
  Your attempt to find an optimal adaptive learning rate at each update step makes sense, however, even the finite difference Newton-Ralphson application for your one-dimensional problem is likely sensitive and unstable numerically since the loss landscape in deep learning is nonconvex (far from quadratic as the classic Newton-Ralphson assumes) even locally. This can lead to learning rate updates unstable. You might try out more stable quasi-Newton methods such as [BFGS](https://en.wikipedia.org/wiki/Broyden%E2%80%93Fletcher%E2%80%93Goldfarb%E2%80%93Shanno_algorithm), but they share the same sensitivity issue for second-order derivatives.
  
  Also if you combine your method with mini-batch like most other deep learning stochastic optimization methods, they're incompatible since the deterministic result of Newton-Ralphson in one mini-batch may not generalize well to the next batch, leading to inconsistent update.
  
  
  ---
  
  