Answers.Body: <p>The most commonly-used optimizer in Deep Reinforcement Learning research the past few years is probably ADAM (or its <a href="http://www.satyenkale.com/papers/amsgrad.pdf" rel="nofollow noreferrer">AMSGrad variant</a>, which in most frameworks like keras/tensorflow/pytorch etc. can be used by setting an <code>amsgrad</code> flag to <code>True</code> in the construction of an ADAM optimizer, and I believe is often also already set to <code>True</code> by default). This is a somewhat newer optimizer which isn't included in that visualization you linked to.</p>
  
  <p>RMSProper is also still quite popular, <a href="https://www.quora.com/Why-does-Google-DeepMind-use-RMSProp-over-Adam-for-RL" rel="nofollow noreferrer">and there may be some arguments that it might be better suited for non-stationary learning problems</a> (which we typically have in RL, our learning targets tend to be non-stationary due to the agent adapting its behaviour over time).</p>
  
  <p>All of these more "fancy" optimizers, typically including some form of momentum term, can in fact be especially useful for optimization in loss "landscapes" that are not smooth. Imagine that your loss landscape looks like a tall mountain, where we'd like our optimizer to "glide" all the way down. Now suppose that the overall trend of a side of the mountain is downwards, but that there are lots of little bumps along the way down, that it's not a smooth mountain. The more fancy optimizers with momentum terms are more likely to "jump" over those bumps than a regular SGD optimizer without momentum terms. This is because, when a little bump is encountered, the local gradient may point the wrong way, but a momentum term "remembers" the overall trend of gliding downwards along the mountain and still points in that same direction for a while.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-01T11:25:58.110Z
Answers.Text: Amsgrad looks good. I'll check it out. But due to search spaces being too large in neural networks, I think imagining local minimas or even saddle points would be incorrect. We will need to go mathematically and identify things. Momentum's problem is that it goes too much into wrong areas which would ruin performance in a dynamic environment.
Answers.UserDisplayName: sagar_acharya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-01T10:00:38.467Z
Answers.LastActivityDate: 2018-12-01T10:00:38.467Z
Answers.UserDisplayName: Dennis Soemers
Body: <p><a href="https://stackoverflow.com/questions/36162180/gradient-descent-vs-adagrad-vs-momentum-in-tensorflow">https://stackoverflow.com/questions/36162180/gradient-descent-vs-adagrad-vs-momentum-in-tensorflow</a></p>
  
  <p>Here, the nice gifs explain how different algorithms approach towards the root. Unfortunately, the environment in the gif is way too simple and real cases have much more complex environments. Also, in reinforcement learning, the solutions should change each moment in a difficult enough environment since things are dynamic.</p>
  
  <p>My question is which optimizer is best for reinforcement learning in such dynamically changing environment? Adadelta should not move beyond local minima so do we have to use SGD or Adadelta with an exploration heuristic? Please let me know in detail your thoughts.</p>
  
Tags: <reinforcement-learning><optimization>
Title: Neural Network Optimizers in Reinforcement Learning non-well behaved environments
UserDisplayName: sagar_acharya
ThreadText: # Neural Network Optimizers in Reinforcement Learning non-well behaved environments
  
  (Asked by: username_0 on 2018-12-01)
  
  <https://stackoverflow.com/questions/36162180/gradient-descent-vs-adagrad-vs-momentum-in-tensorflow>
  
  Here, the nice gifs explain how different algorithms approach towards the root. Unfortunately, the environment in the gif is way too simple and real cases have much more complex environments. Also, in reinforcement learning, the solutions should change each moment in a difficult enough environment since things are dynamic.
  
  My question is which optimizer is best for reinforcement learning in such dynamically changing environment? Adadelta should not move beyond local minima so do we have to use SGD or Adadelta with an exploration heuristic? Please let me know in detail your thoughts.
  
  **Tags:** reinforcement-learning, optimization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-12-01. Score: 2 | Accepted answer
  
  The most commonly-used optimizer in Deep Reinforcement Learning research the past few years is probably ADAM (or its [AMSGrad variant](http://www.satyenkale.com/papers/amsgrad.pdf), which in most frameworks like keras/tensorflow/pytorch etc. can be used by setting an `amsgrad` flag to `True` in the construction of an ADAM optimizer, and I believe is often also already set to `True` by default). This is a somewhat newer optimizer which isn't included in that visualization you linked to.
  
  RMSProper is also still quite popular, [and there may be some arguments that it might be better suited for non-stationary learning problems](https://www.quora.com/Why-does-Google-DeepMind-use-RMSProp-over-Adam-for-RL) (which we typically have in RL, our learning targets tend to be non-stationary due to the agent adapting its behaviour over time).
  
  All of these more "fancy" optimizers, typically including some form of momentum term, can in fact be especially useful for optimization in loss "landscapes" that are not smooth. Imagine that your loss landscape looks like a tall mountain, where we'd like our optimizer to "glide" all the way down. Now suppose that the overall trend of a side of the mountain is downwards, but that there are lots of little bumps along the way down, that it's not a smooth mountain. The more fancy optimizers with momentum terms are more likely to "jump" over those bumps than a regular SGD optimizer without momentum terms. This is because, when a little bump is encountered, the local gradient may point the wrong way, but a momentum term "remembers" the overall trend of gliding downwards along the mountain and still points in that same direction for a while.
  
  > **Comments:**
  > **username_0**: Amsgrad looks good. I'll check it out. But due to search spaces being too large in neural networks, I think imagining local minimas or even saddle points would be incorrect. We will need to go mathematically and identify things. Momentum's problem is that it goes too much into wrong areas which would ruin performance in a dynamic environment.  
  
  ---
  
  