Answers.Body: <p>Great question! I did some research and found out that <a href="https://deepmind.com/blog/article/generally-capable-agents-emerge-from-open-ended-play" rel="nofollow noreferrer">Generally capable agents emerge from open ended play</a> by Deepmind is using ES in form of population based training:</p>
  <blockquote>
  <p>We also explored the question, what distribution of training tasks will produce the best possible agent, especially in such a vast environment? The dynamic task generation we use allows for continual changes to the distribution of the agent’s training tasks: every task is generated to be neither too hard nor too easy, but just right for training. We then use population based training (PBT) to adjust the parameters of the dynamic task generation based on a fitness that aims to improve agents’ general capability. And finally we chain together multiple training runs so each generation of agents can bootstrap off the previous generation.</p>
  </blockquote>
  <p>But this didn't really answer on their reasoning, so I dug some deeper and found a <a href="https://www.lesswrong.com/posts/DreKBuMvK7fdESmSJ/how-deepmind-s-generally-capable-agents-were-trained#3__Population_Based_Training" rel="nofollow noreferrer">great article on lesswrong.com about the use of PBT</a>. I will quote the essence, but highly recommend to read the linked Chapter on PBT:</p>
  <blockquote>
  <p>What does the evolutionary selection give us that we don't already
  have? What problem does this let us avoid?</p>
  <p>There are several answers to this question.</p>
  <p>The more narrow answer is that this allows the dynamic task generation
  hyper parameters themselves to shift in a direction that promotes
  general competence. Neither of the optimization levels beneath us
  include any way of changing these parameters. But the ideal filtering
  parameters for the production of general competence might be different
  at the beginning, or at the middle of training. Or they might be
  different from agent to agent. Without something like
  population-based-training, they would have no way of changing and this
  would hurt performance.</p>
  <p>The less narrow answer, I think, is that this ensures that agents are
  developing broad competence in a way the innermost loop cannot do.
  [...]  each agent in our population of agents will learn to get better at
  some distribution of tasks, then, but without
  population-based-training they might not spread themselves broadly
  across the entire span of this distribution. Like a student who
  advances wildly at subjects she prefers, while ignoring subjects she
  is not good at, our agents might not approach our desired ideal of
  general competence. Population-based-training helps prevent the
  scenario by multiplying agent / teacher pairs that do well generally
  and non-narrowly.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-04T16:40:40.437Z
Answers.LastActivityDate: 2021-09-04T16:40:40.437Z
Answers.UserDisplayName: tnfru
Body: <p>In <a href="https://arxiv.org/abs/1703.03864" rel="noreferrer">Salimans et al, 2016</a>, the authors argue that ES should be considered a competitive alternative to MDP-based RL algorithms like Q-Learning, TRPO.</p>
  <p>However, in practice, I notice that more often than not ES takes far more episodes to converge than MDP-based algorithms. So what would still be a reason to consider those, apart from pure academic interest?</p>
  <p>The authors mention that ES will show less variance in long-horizon tasks, but didn't give an example. Is this aspect crucial ?</p>
  
Tags: <reinforcement-learning><evolutionary-algorithms>
Title: When would you use Evolutionary Strategies over Step-Based Reinforcement Learning
UserDisplayName: ganto
ThreadText: # When would you use Evolutionary Strategies over Step-Based Reinforcement Learning
  
  (Asked by: username_0 on 2021-09-01)
  
  In [Salimans et al, 2016](https://arxiv.org/abs/1703.03864), the authors argue that ES should be considered a competitive alternative to MDP-based RL algorithms like Q-Learning, TRPO.
  
  However, in practice, I notice that more often than not ES takes far more episodes to converge than MDP-based algorithms. So what would still be a reason to consider those, apart from pure academic interest?
  
  The authors mention that ES will show less variance in long-horizon tasks, but didn't give an example. Is this aspect crucial ?
  
  **Tags:** reinforcement-learning, evolutionary-algorithms
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-09-04. Score: 1
  
  Great question! I did some research and found out that [Generally capable agents emerge from open ended play](https://deepmind.com/blog/article/generally-capable-agents-emerge-from-open-ended-play) by Deepmind is using ES in form of population based training:
  
  > We also explored the question, what distribution of training tasks will produce the best possible agent, especially in such a vast environment? The dynamic task generation we use allows for continual changes to the distribution of the agent’s training tasks: every task is generated to be neither too hard nor too easy, but just right for training. We then use population based training (PBT) to adjust the parameters of the dynamic task generation based on a fitness that aims to improve agents’ general capability. And finally we chain together multiple training runs so each generation of agents can bootstrap off the previous generation.
  
  But this didn't really answer on their reasoning, so I dug some deeper and found a [great article on lesswrong.com about the use of PBT](https://www.lesswrong.com/posts/DreKBuMvK7fdESmSJ/how-deepmind-s-generally-capable-agents-were-trained#3__Population_Based_Training). I will quote the essence, but highly recommend to read the linked Chapter on PBT:
  
  > What does the evolutionary selection give us that we don't already
  > have? What problem does this let us avoid?
  >
  > There are several answers to this question.
  >
  > The more narrow answer is that this allows the dynamic task generation
  > hyper parameters themselves to shift in a direction that promotes
  > general competence. Neither of the optimization levels beneath us
  > include any way of changing these parameters. But the ideal filtering
  > parameters for the production of general competence might be different
  > at the beginning, or at the middle of training. Or they might be
  > different from agent to agent. Without something like
  > population-based-training, they would have no way of changing and this
  > would hurt performance.
  >
  > The less narrow answer, I think, is that this ensures that agents are
  > developing broad competence in a way the innermost loop cannot do.
  > [...] each agent in our population of agents will learn to get better at
  > some distribution of tasks, then, but without
  > population-based-training they might not spread themselves broadly
  > across the entire span of this distribution. Like a student who
  > advances wildly at subjects she prefers, while ignoring subjects she
  > is not good at, our agents might not approach our desired ideal of
  > general competence. Population-based-training helps prevent the
  > scenario by multiplying agent / teacher pairs that do well generally
  > and non-narrowly.
  
  
  ---
  
  