Answers.Body: <p>We usually optimize <strong>with respect to</strong> something. For example, you can train a neural network to locate cats in an image. This operation of locating cats in an image can be thought of as a function: given an image, a neural network can be trained to return the position of the cat in the image. In this sense, we can optimize a neural network with respect to this task. </p>
  
  <p>However, if a neural network represents an optimization algorithm, then, if you change it a little bit, then it will no more be the same optimization algorithm: it might be another optimization algorithm or some other different algorithm. </p>
  
  <p>For example, most optimizations algorithms that are used to train neural networks (like Adam) are a variation of gradient descent (GD). If you think that Adam performs better than GD, then you could say that Adam is an optimization of GD. So, Adam performs better than GD with respect to something. Possibly, GD also performs better than Adam with respect to something else. Of course, this is a little bit of a stretch.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-23T22:29:29.700Z
Answers.LastActivityDate: 2019-07-23T22:29:29.700Z
Answers.UserDisplayName: nbro
Answers.Body: <p>That does not seem very useful to apply local minima search (as SGD) to another local minima search. Existing successful solutions combine global minima search techniques with local minima search.</p>
  
  <p>For example, it's beneficial to combine simulated annealing with SGD to optimize it's learning rate and/or Nesterov momentum. In this case, you don't even need to spawn a population of SGD optimizers. But, you can also try population-based algorithms like evolution programming.</p>
  
  <p>The idea to optimize optimizers is very curious, but it's rather useful to try it on global optimization algorithms.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-25T22:54:57.010Z
Answers.Text: Could you please provide a link to a research work/paper where this has been done: "it's beneficial to combine simulated annealing with SGD"?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-13T19:48:30.960Z
Answers.LastActivityDate: 2020-05-13T19:48:30.960Z
Answers.UserDisplayName: Bogdan Ruzhitskiy
Answers.Body: <p>First, you need to consider what are the "parameters" of this "optimization algorithm" that you want to "optimize". Let's take the most simple case, a SGD without momentum. The update rule for this optimizer is:</p>
  
  <p><span class="math-container">$$
  w_{t+1} \leftarrow w_{t} - a \cdot \nabla_{w_{t}} J(w_t) = w_{t} - a \cdot g_t
  $$</span></p>
  
  <p>where <span class="math-container">$w_t$</span> are the weights at iteration <span class="math-container">$t$</span>, <span class="math-container">$J$</span> is the cost function, <span class="math-container">$g_t = \nabla_{w_{t}} J(w_t)$</span> are the gradients of the cost function w.r.t <span class="math-container">$w_t$</span> and <span class="math-container">$a$</span> is the learning rate.</p>
  
  <p>An optimization algorithm accepts as its input the weights and their gradients and returns the update. So we could write the above equation as:</p>
  
  <p><span class="math-container">$$
  w_{t+1} \leftarrow w_{t} - SGD(w_t, g_t)
  $$</span></p>
  
  <p>The same is true for all optimization algorithms (e.g. Adam, RMSprop, etc.). Now our initial question was what are the <strong>parameters of the optimizer</strong>, which we want to optimize. In the simple case of the SGD, the sole parameter of the optimizer is the <strong>learning rate</strong>. </p>
  
  <p>The question that arises at this point is <em>can we optimize the learning rate of the optimizer during training?</em> Or more practically, can we compute this derivative?</p>
  
  <p><span class="math-container">$$
  \frac{\partial J(w_t)}{\partial a}
  $$</span></p>
  
  <p>This idea was explored in <a href="https://arxiv.org/pdf/1703.04782.pdf" rel="noreferrer">this paper</a>, where they coin this technique "hypergradient descent". I suggest you take a look.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-28T12:13:46.667Z
Answers.Text: In particular, one would expect an 'optimised algorithm' not to need any external parameters to be chosen.
Answers.UserDisplayName: Andre Holzner
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-25T22:56:54.823Z
Answers.Text: It would be nice if you at least briefly described the idea of computing the derivative with respect to the learning rate proposed in that paper. I suppose that the loss function is also a function of the learning rate, and that's how they do it, otherwise, right now, I'm not seeing how they could do that.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-23T22:28:27.030Z
Answers.LastActivityDate: 2019-07-23T22:28:27.030Z
Answers.UserDisplayName: Djib2011
Body: <p>In <a href="https://ai.stackexchange.com/a/13543/2444">this answer</a> to the question <a href="https://ai.stackexchange.com/q/13524/2444">Is an optimization algorithm equivalent to a neural network?</a>, the author stated that, in theory, there is some recurrent neural network that implements a given optimization algorithm.</p>
  <p>If so, then can we optimize the optimization algorithm?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-12T09:11:56.263Z
Comments.Text: [Here](https://ai.stackexchange.com/q/32553/2444) is a related question.
Comments.UserDisplayName: nbro
Tags: <neural-networks><meta-learning>
Title: Can we optimize an optimization algorithm?
UserDisplayName: Dimer
ThreadText: # Can we optimize an optimization algorithm?
  
  (Asked by: username_0 on 2019-07-23)
  
  In [this answer](https://ai.stackexchange.com/a/13543/2444) to the question [Is an optimization algorithm equivalent to a neural network?](https://ai.stackexchange.com/q/13524/2444), the author stated that, in theory, there is some recurrent neural network that implements a given optimization algorithm.
  
  If so, then can we optimize the optimization algorithm?
  
  **Tags:** neural-networks, meta-learning
  
  > **Comments:**
  > **username_1**: [Here](https://ai.stackexchange.com/q/32553/2444) is a related question.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2019-07-23. Score: 5 | Accepted answer
  
  First, you need to consider what are the "parameters" of this "optimization algorithm" that you want to "optimize". Let's take the most simple case, a SGD without momentum. The update rule for this optimizer is:
  
  $$
  w\_{t+1} \leftarrow w\_{t} - a \cdot \nabla\_{w\_{t}} J(w\_t) = w\_{t} - a \cdot g\_t
  $$
  
  where $w\_t$ are the weights at iteration $t$, $J$ is the cost function, $g\_t = \nabla\_{w\_{t}} J(w\_t)$ are the gradients of the cost function w.r.t $w\_t$ and $a$ is the learning rate.
  
  An optimization algorithm accepts as its input the weights and their gradients and returns the update. So we could write the above equation as:
  
  $$
  w\_{t+1} \leftarrow w\_{t} - SGD(w\_t, g\_t)
  $$
  
  The same is true for all optimization algorithms (e.g. Adam, RMSprop, etc.). Now our initial question was what are the **parameters of the optimizer**, which we want to optimize. In the simple case of the SGD, the sole parameter of the optimizer is the **learning rate**.
  
  The question that arises at this point is *can we optimize the learning rate of the optimizer during training?* Or more practically, can we compute this derivative?
  
  $$
  \frac{\partial J(w\_t)}{\partial a}
  $$
  
  This idea was explored in [this paper](https://arxiv.org/pdf/1703.04782.pdf), where they coin this technique "hypergradient descent". I suggest you take a look.
  
  > **Comments:**
  > **username_4**: In particular, one would expect an 'optimised algorithm' not to need any external parameters to be chosen.  
  > **username_1**: It would be nice if you at least briefly described the idea of computing the derivative with respect to the learning rate proposed in that paper. I suppose that the loss function is also a function of the learning rate, and that's how they do it, otherwise, right now, I'm not seeing how they could do that.  
  
  ---
  
  ### Answer by username_1 on 2019-07-23. Score: 2
  
  We usually optimize **with respect to** something. For example, you can train a neural network to locate cats in an image. This operation of locating cats in an image can be thought of as a function: given an image, a neural network can be trained to return the position of the cat in the image. In this sense, we can optimize a neural network with respect to this task.
  
  However, if a neural network represents an optimization algorithm, then, if you change it a little bit, then it will no more be the same optimization algorithm: it might be another optimization algorithm or some other different algorithm.
  
  For example, most optimizations algorithms that are used to train neural networks (like Adam) are a variation of gradient descent (GD). If you think that Adam performs better than GD, then you could say that Adam is an optimization of GD. So, Adam performs better than GD with respect to something. Possibly, GD also performs better than Adam with respect to something else. Of course, this is a little bit of a stretch.
  
  
  ---
  
  ### Answer by username_2 on 2020-05-13. Score: 0
  
  That does not seem very useful to apply local minima search (as SGD) to another local minima search. Existing successful solutions combine global minima search techniques with local minima search.
  
  For example, it's beneficial to combine simulated annealing with SGD to optimize it's learning rate and/or Nesterov momentum. In this case, you don't even need to spawn a population of SGD optimizers. But, you can also try population-based algorithms like evolution programming.
  
  The idea to optimize optimizers is very curious, but it's rather useful to try it on global optimization algorithms.
  
  > **Comments:**
  > **username_1**: Could you please provide a link to a research work/paper where this has been done: "it's beneficial to combine simulated annealing with SGD"?  
  
  ---
  
  