Answers.Body: <p>When it comes to genetic algorithms (neuroevolution), you can pretty much evolve any kind of parameters. You just need to have a fairly complex system where each parameter changes the way the inputs are affecting the outputs.</p>
  
  <p>So, to answer your question, any activation function should work! It would be surprising that the activation function is your problem. But if you want to test some activation functions, here's a <a href="https://en.wikipedia.org/wiki/Activation_function" rel="nofollow noreferrer">list of activation functions</a>. But tanh should work perfectly fine.</p>
  
  <p>As I already mentioned in another answer, the key to have a great neuroevolution algorithm is the fitness function. Your rewards should reflect the goal you're aiming for. You didn't mentioned what fitness function you used, but here's what I suggest: reward a player ONLY if he progresses in the game. For example, your fitness function could simply be the distance traveled by your birds. This could maybe solve your problem...</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-04T02:51:04.167Z
Answers.LastActivityDate: 2020-05-02T15:32:51.210Z
Answers.LastEditDate: 2020-05-02T15:32:51.210Z
Answers.UserDisplayName: Dara Kong
Body: <p>I am developing a NEAT flappy bird game, and it doesn't work, the system stays stupid for 300 generations. I chose tanh() for activation, just because it's included in JS.</p>
  
  <p>I can't find a good discussion on the internet of activation functions in the context of neuroevolution, most of what I see is about derivative and other gradient descent issues which I suspect are irrelevant to forward only networks.</p>
  
  <p>If you need a fixed point to answer, I have 8 inputs, one output and the problem is a classification ("jump", "don't jump"). But please explain your answer. I currently use tanh() for all the hidden and output nodes, and the output is considered "jump" if the output neuron value is >0.85</p>
  
  <p>For some context, the code is here: <a href="https://github.com/nraynaud/nraygame" rel="nofollow noreferrer">https://github.com/nraynaud/nraygame</a> and the game here: <a href="https://nraynaud.github.io/nraygame/" rel="nofollow noreferrer">https://nraynaud.github.io/nraygame/</a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-02T15:37:29.890Z
Comments.Text: @nraynaud Did you just fixed your problem? Because I tested your game, and the birds are really good at the after only 30 generations. If you found the solution to your problem, you can answer your own question to help others who have the same question.
Comments.UserDisplayName: Dara Kong
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-03T16:22:56.453Z
Comments.Text: @DaraKong Sadly no, I never could make it understand the link between pipe2 and pipeX. It's still always looking at the leftmost pipe even when it's already behind the bird. When the game becomes hard, the birds stay stupid.
Comments.UserDisplayName: nraynaud
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-03T20:48:42.067Z
Comments.Text: @nraynaud I don't know why, but when every time I go to your game's url you gave us in your question, the agents are performing very well after only 25 generations. For example : generation 26 survived to time 50000, and the graph was showing that your birds were actually learning fast. I also tried to download your code, and test it locally. Everything was looking fine, and the agents were as good as in the online version. I can't figure out why you are experiencing problems, the results are supposed to be the same right ? What are your results precisely ? What is the best time you got ?
Comments.UserDisplayName: Dara Kong
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-04T22:02:31.450Z
Comments.Text: thank you very much for your interest. Over time the game gets harder and harder, the pipes gets closer, and there is a point where the system will have to look at the second pipe before the first one disappears on the left if it doesn't want to die. I am trying to make it evolve a behavior smarter than just "match the height of the leftmost pipe".
Comments.UserDisplayName: nraynaud
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-24T12:24:55.327Z
Comments.Text: Why do you use only 1 output neuron? Why not use 2 output neurons? 1 for jump and 1 for not jump?
Comments.UserDisplayName: Lustwelpintje
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-24T12:39:12.490Z
Comments.Text: it feels more binary to have number and a threshold, and simpler to have less nodes, but if you have a good explanation as to why 2 is better than one, I just want to learn.
Comments.UserDisplayName: nraynaud
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-24T13:49:30.233Z
Comments.Text: I don't have many experience in NEAT, but I have made some reinforcement learning NNs. When you are training your NN using the MDP logic,Your output is the value of each action. If you have 2 output neurons this would be the value of jumping and the value of not jumping. when comparing these 2 values, you can then take the action with the biggest number. When you only have one output neuron, you can't train these values.However, when you use supervised learning instead of reinforcement learning, you can use your method (because the system learns from your predefined data, not from experience).
Comments.UserDisplayName: Lustwelpintje
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-24T13:52:45.337Z
Comments.Text: I just checked, NEAT works with reinforcement learning methods. So using 2 output neurons could make it learn better.
Comments.UserDisplayName: Lustwelpintje
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-24T14:15:46.510Z
Comments.Text: thanks, that could be a teaching moment, I'm googling this vocabulary.
Comments.UserDisplayName: nraynaud
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-24T16:21:08.973Z
Comments.Text: What is the difference between 2 output neurons whose value is compared in the code and having one output neuron and 2 hiddens neurons right before it, whose output would be combined with a negative sign in the NN? Shouldn't the system discover this structure?
Comments.UserDisplayName: nraynaud
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-25T10:24:48.507Z
Comments.Text: Reinforcement learning works by training the network with the values of the actions. If you have 1 output neuron you will train that output neuron to be higher if you got far and lower if you died immediatly. In this case, it will be trained randomly. If you have 2 output neurons, you will do a run, select a action, and train 1 of those output neurons (which will be the action you selected). This way, the other output neuron will be unaffected. In your way, the network won't learn the 2 preceding hidden neurons the way you want to, because it is trained on the reward on the next output neuron.
Comments.UserDisplayName: Lustwelpintje
Tags: <neat><activation-functions><neuroevolution>
Title: How to choose the activation function in neuroevolution?
UserDisplayName: nraynaud
ThreadText: # How to choose the activation function in neuroevolution?
  
  (Asked by: username_0 on 2020-02-24)
  
  I am developing a NEAT flappy bird game, and it doesn't work, the system stays stupid for 300 generations. I chose tanh() for activation, just because it's included in JS.
  
  I can't find a good discussion on the internet of activation functions in the context of neuroevolution, most of what I see is about derivative and other gradient descent issues which I suspect are irrelevant to forward only networks.
  
  If you need a fixed point to answer, I have 8 inputs, one output and the problem is a classification ("jump", "don't jump"). But please explain your answer. I currently use tanh() for all the hidden and output nodes, and the output is considered "jump" if the output neuron value is >0.85
  
  For some context, the code is here: <https://github.com/nraynaud/nraygame> and the game here: <https://nraynaud.github.io/nraygame/>
  
  **Tags:** neat, activation-functions, neuroevolution
  > **Comments:**
  > **username_2**: Why do you use only 1 output neuron? Why not use 2 output neurons? 1 for jump and 1 for not jump?  
  > **username_0**: it feels more binary to have number and a threshold, and simpler to have less nodes, but if you have a good explanation as to why 2 is better than one, I just want to learn.  
  > **username_2**: I don't have many experience in NEAT, but I have made some reinforcement learning NNs. When you are training your NN using the MDP logic,Your output is the value of each action. If you have 2 output neurons this would be the value of jumping and the value of not jumping. when comparing these 2 values, you can then take the action with the biggest number. When you only have one output neuron, you can't train these values.However, when you use supervised learning instead of reinforcement learning, you can use your method (because the system learns from your predefined data, not from experience).  
  > **username_2**: I just checked, NEAT works with reinforcement learning methods. So using 2 output neurons could make it learn better.  
  > **username_0**: thanks, that could be a teaching moment, I'm googling this vocabulary.  
  > **username_0**: What is the difference between 2 output neurons whose value is compared in the code and having one output neuron and 2 hiddens neurons right before it, whose output would be combined with a negative sign in the NN? Shouldn't the system discover this structure?  
  > **username_2**: Reinforcement learning works by training the network with the values of the actions. If you have 1 output neuron you will train that output neuron to be higher if you got far and lower if you died immediatly. In this case, it will be trained randomly. If you have 2 output neurons, you will do a run, select a action, and train 1 of those output neurons (which will be the action you selected). This way, the other output neuron will be unaffected. In your way, the network won't learn the 2 preceding hidden neurons the way you want to, because it is trained on the reward on the next output neuron.  
  > **username_1**: @nraynaud Did you just fixed your problem? Because I tested your game, and the birds are really good at the after only 30 generations. If you found the solution to your problem, you can answer your own question to help others who have the same question.  
  > **username_0**: @DaraKong Sadly no, I never could make it understand the link between pipe2 and pipeX. It's still always looking at the leftmost pipe even when it's already behind the bird. When the game becomes hard, the birds stay stupid.  
  > **username_1**: @nraynaud I don't know why, but when every time I go to your game's url you gave us in your question, the agents are performing very well after only 25 generations. For example : generation 26 survived to time 50000, and the graph was showing that your birds were actually learning fast. I also tried to download your code, and test it locally. Everything was looking fine, and the agents were as good as in the online version. I can't figure out why you are experiencing problems, the results are supposed to be the same right ? What are your results precisely ? What is the best time you got ?  
  > **username_0**: thank you very much for your interest. Over time the game gets harder and harder, the pipes gets closer, and there is a point where the system will have to look at the second pipe before the first one disappears on the left if it doesn't want to die. I am trying to make it evolve a behavior smarter than just "match the height of the leftmost pipe".  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-04-04. Score: 1
  
  When it comes to genetic algorithms (neuroevolution), you can pretty much evolve any kind of parameters. You just need to have a fairly complex system where each parameter changes the way the inputs are affecting the outputs.
  
  So, to answer your question, any activation function should work! It would be surprising that the activation function is your problem. But if you want to test some activation functions, here's a [list of activation functions](https://en.wikipedia.org/wiki/Activation_function). But tanh should work perfectly fine.
  
  As I already mentioned in another answer, the key to have a great neuroevolution algorithm is the fitness function. Your rewards should reflect the goal you're aiming for. You didn't mentioned what fitness function you used, but here's what I suggest: reward a player ONLY if he progresses in the game. For example, your fitness function could simply be the distance traveled by your birds. This could maybe solve your problem...
  
  
  ---
  
  