Answers.Body: <p>In my opinion, this shouldn't be an either/or question.  Both NEAT and rl with fixed network topology has their own advantage when solving decision problem.</p>
  <p>NEAT is good to solve a simple problem fast with minimum network topology and without local optimum issue.
  While RL with fixed topology suffers local optimum but learning more directly, and policy-gradient based Rl would be even more adaptive to an environment with stochasticity in the context of statistics.</p>
  <p>Then, why not just combine them? At present, the best paper on this topic is this： <a href="https://dl.acm.org/doi/10.1145/3205455.3205536" rel="nofollow noreferrer">https://dl.acm.org/doi/10.1145/3205455.3205536</a>
  In this paper, it is suggested that doing NEAT at first, then KEEP DOING RL forever, though I don't agree with this special routine.
  I think, both NEAT and RL should do interactively during the WHOLE training process.</p>
  <p>The problem is, how to combine these 2 in an effective way. One problem I met is for RL, like SAC, which has 2 outputs(one for policy and one for Q-value), And the topology of the Q-value output has no contribution when doing NEAT. Then, how to deal with the Q-value topology? If both the 2 outputs shares some layers at first, then the learning would become more unstable, since doing NEAT would dramatically change these 2 outputs.</p>
  <p>The paper above takes a fixed output layer for both 2 outputs, which I think is just a workaround, not the best way.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-03T14:41:24.327Z
Answers.LastActivityDate: 2022-10-24T00:52:06.140Z
Answers.LastEditDate: 2022-10-24T00:52:06.140Z
Answers.UserDisplayName: Junwei Dong
Body: <p>NEAT is an evolutionary algorithm. When would you want to use NEAT over more traditional/common RL algorithms like PPO or SAC etc. What advantage does it give you?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-13T14:31:07.800Z
Comments.Text: It seems that you know what NEAT can be used for. Do you know what RL is used for? If yes, can you give an example of when you think you could use NEAT rather than RL (which doesn't mean it is necessarily a better approach than RL)?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-13T17:10:04.050Z
Comments.Text: Sure, in the NEAT python repo they have examples of applying NEAT to OpenAI lunar-lander and cartpole problems. Are you asking a rhetorical question (actually just asking so i dont sound dumb lol)?
Comments.UserDisplayName: Dylan Kerler
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-13T20:16:14.607Z
Comments.Text: I was just making sure that you know what RL is used for, and if you know how NEAT could be applied to RL problems. Maybe you can briefly state in your post what NEAT can be used for. To find a policy? To evolve neural networks that represent the policy that you want to find in RL?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-13T21:31:39.547Z
Comments.Text: Yes, I'm very familiar with the de-facto RL like using PPO, Q-Learning etc. NEAT can be used to find a policy through "evolution" of both the neural net architecture and the weights in the neural net. I'm wondering in what situation would NEAT be better than policy gradient RL.
Comments.UserDisplayName: Dylan Kerler
Tags: <machine-learning><reinforcement-learning><deep-rl><neat>
Title: In what situation would you want to use NEAT over reinforcement learning?
UserDisplayName: Dylan Kerler
ThreadText: # In what situation would you want to use NEAT over reinforcement learning?
  
  (Asked by: username_0 on 2021-08-12)
  
  NEAT is an evolutionary algorithm. When would you want to use NEAT over more traditional/common RL algorithms like PPO or SAC etc. What advantage does it give you?
  
  **Tags:** machine-learning, reinforcement-learning, deep-rl, neat
  > **Comments:**
  > **username_1**: It seems that you know what NEAT can be used for. Do you know what RL is used for? If yes, can you give an example of when you think you could use NEAT rather than RL (which doesn't mean it is necessarily a better approach than RL)?  
  > **username_0**: Sure, in the NEAT python repo they have examples of applying NEAT to OpenAI lunar-lander and cartpole problems. Are you asking a rhetorical question (actually just asking so i dont sound dumb lol)?  
  > **username_1**: I was just making sure that you know what RL is used for, and if you know how NEAT could be applied to RL problems. Maybe you can briefly state in your post what NEAT can be used for. To find a policy? To evolve neural networks that represent the policy that you want to find in RL?  
  > **username_0**: Yes, I'm very familiar with the de-facto RL like using PPO, Q-Learning etc. NEAT can be used to find a policy through "evolution" of both the neural net architecture and the weights in the neural net. I'm wondering in what situation would NEAT be better than policy gradient RL.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-04-03. Score: 1
  
  In my opinion, this shouldn't be an either/or question. Both NEAT and rl with fixed network topology has their own advantage when solving decision problem.
  
  NEAT is good to solve a simple problem fast with minimum network topology and without local optimum issue.
  While RL with fixed topology suffers local optimum but learning more directly, and policy-gradient based Rl would be even more adaptive to an environment with stochasticity in the context of statistics.
  
  Then, why not just combine them? At present, the best paper on this topic is this： <https://dl.acm.org/doi/10.1145/3205455.3205536>
  In this paper, it is suggested that doing NEAT at first, then KEEP DOING RL forever, though I don't agree with this special routine.
  I think, both NEAT and RL should do interactively during the WHOLE training process.
  
  The problem is, how to combine these 2 in an effective way. One problem I met is for RL, like SAC, which has 2 outputs(one for policy and one for Q-value), And the topology of the Q-value output has no contribution when doing NEAT. Then, how to deal with the Q-value topology? If both the 2 outputs shares some layers at first, then the learning would become more unstable, since doing NEAT would dramatically change these 2 outputs.
  
  The paper above takes a fixed output layer for both 2 outputs, which I think is just a workaround, not the best way.
  
  
  ---
  
  