Answers.Body: <p>The most general descriptive frameworks covering what you are trying to do are:</p>
  <ul>
  <li><p><a href="https://en.wikipedia.org/wiki/Sequential_decision_making" rel="nofollow noreferrer">Sequential decision making</a> (article is a stub, but the term a good launching point to discover different wys of modelling and solving these kind of problems)</p>
  </li>
  <li><p><a href="https://en.wikipedia.org/wiki/Optimal_control" rel="nofollow noreferrer">Optimal control</a></p>
  </li>
  </ul>
  <p>These put some context around your problem, and might give you some pointers. For instance, reinforcement learning is an alternative approach to the evolutionary system you are trying to build.</p>
  <p>The specific AI system you appear to be building is a <a href="https://en.wikipedia.org/wiki/Genetic_algorithm" rel="nofollow noreferrer">genetic algorithm</a>, and more specific still you are attempting to find a neural network that is optimal at a task by searching for the best network using a system of population generation, selection and mutation which repeats.</p>
  <p>There are lots of ways to set up a system like this, so your approach is not necessarily <em>wrong</em>. However, I think there are two key things that would improve what you have built so far:</p>
  <ul>
  <li><p>Use a <a href="https://en.wikipedia.org/wiki/Fitness_function" rel="nofollow noreferrer">fitness function</a> for selection. Score each car, perhaps by how far it got before crashing when the episode ends. To reduce luck factor on random courses, you could make this score the mean result from e.g. 3 different courses (it is not necessary, but may address your concern that selection is too random in your case). Select some fraction of top scoring cars, or look into other selection approaches - e.g. weighted selection based on fitness score or ranking.</p>
  </li>
  <li><p>Add &quot;sex&quot;, more properly known as genome <a href="https://en.wikipedia.org/wiki/Crossover_(genetic_algorithm)" rel="nofollow noreferrer">crossover</a> between selected population members. Mutating individuals is limiting because it silos improvements to a single line of ancestry - if there are two good mutations found at random you rely on that single line finding both of them. Whilst crossover allows sharing of good mutations between lines, making it much more likely that two good mutations will end up in the same individual.</p>
  </li>
  </ul>
  <p>There is a framework called <a href="https://en.wikipedia.org/wiki/Neuroevolution_of_augmenting_topologies" rel="nofollow noreferrer">NEAT</a> which covers the issues above plus has other features useful for evolving neural networks. It often does well at control scenarios like the one you are considering. You may want to look into it, if your focus is mainly on solving the control problem. However, it is relatively advanced from where you are, so if your current focus to learn by building from scratch you may get more initially from implementing fitness functions and crossover yourself.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-18T15:02:04.413Z
Answers.Text: thank you very much. I've seen this NEAT in the past but didn't understand it. Now that i played with my simple NN i think i understand it. (and i think that the game called "GridWorld" is doing this). My NN have a fixed topology (defined by me : 7 raycasting input, 1 speed input, 2 hidden layer of 12 and 6 neuron, 2 for speed control and direction) and i'm not encoding phenotype and therefore make crossover more difficult in my code. But i will try this from now on. i could even let the NN try different input sensor by itself (ex : angle of the raycast, tyre friction (drifing in turn), etc)
Answers.UserDisplayName: ker2x
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-18T15:16:05.300Z
Answers.Text: It's ambitious but worth it i guess. i won't go anywhere if i keep playing with the same primitive NN forever and it's totally in the direction toward my end goal : a mini-game where you watch a NN learning, playing failing, evolving, ... i specifically don't want a sophisticated fitness function or error computation : it's either "win the right to reproduce/mutate" or "die without offspring". And neat seems to fit the need.
Answers.UserDisplayName: ker2x
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-18T12:10:11.903Z
Answers.LastActivityDate: 2020-11-18T12:10:11.903Z
Answers.UserDisplayName: Neil Slater
Body: <p>I'm a beginner with a classic &quot;racing car&quot; sandbox and a homemade simple neural network.</p>
  <p>My pattern:</p>
  <ol>
  <li><p>Copy the &quot;top car&quot; (without mutation) to the next generation</p>
  </li>
  <li><p>If there are some cars still running (because simulation reached the 30s win condition), then copy a mutated version of them for the next generation.</p>
  </li>
  <li><p>Fill the rest of the pool with mutation of the &quot;top car&quot;.</p>
  </li>
  </ol>
  <p>But this is just some dumb intuitive pattern I made on the fly while playing with my code. Perhaps I should copy the cars that are still running as-is instead of mutating them. Or, perhaps, some selection method I don't know about.</p>
  <p>A new random track is generated at each new generation. a &quot;top car&quot; may be good on a track and crash immediately on the next track. I just feel that basing everything on the top car is wrong because of the track randomness.</p>
  <p><strong>Is there some known pattern for selecting a batch of candidates?</strong> (paper, google-fu keyword, interesting blog, etc.)</p>
  <p>I don't know what to search for. I don't even know the name of my network or any vocabulary related to AI.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-18T12:36:32.140Z
Comments.Text: What does your neural network represent? It's also not clear to me what you're trying to do. Are you trying to find the best car (that is able to drive in some environment: please, which specify which environment) by having a population of cars that you evolve my mutating? That seems to be just a **genetic algorithm**.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-18T12:54:37.330Z
Comments.Text: my post was originall post was bigger and explained what you want but i was told i wasn''t concise enough :D
Comments.UserDisplayName: ker2x
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-18T13:31:30.483Z
Comments.Text: I only asked you to ask one question, because you had multiple questions. I didn't say you should remove relevant details. Anyway, you can edit your post again.
Comments.UserDisplayName: nbro
Tags: <neural-networks><reference-request><genetic-algorithms><genetic-operators>
Title: Is there some known pattern for selecting a batch of candidates for the next generation?
UserDisplayName: ker2x
ThreadText: # Is there some known pattern for selecting a batch of candidates for the next generation?
  
  (Asked by: username_0 on 2020-11-18)
  
  I'm a beginner with a classic "racing car" sandbox and a homemade simple neural network.
  
  My pattern:
  
  1. Copy the "top car" (without mutation) to the next generation
  2. If there are some cars still running (because simulation reached the 30s win condition), then copy a mutated version of them for the next generation.
  3. Fill the rest of the pool with mutation of the "top car".
  
  But this is just some dumb intuitive pattern I made on the fly while playing with my code. Perhaps I should copy the cars that are still running as-is instead of mutating them. Or, perhaps, some selection method I don't know about.
  
  A new random track is generated at each new generation. a "top car" may be good on a track and crash immediately on the next track. I just feel that basing everything on the top car is wrong because of the track randomness.
  
  **Is there some known pattern for selecting a batch of candidates?** (paper, google-fu keyword, interesting blog, etc.)
  
  I don't know what to search for. I don't even know the name of my network or any vocabulary related to AI.
  
  **Tags:** neural-networks, reference-request, genetic-algorithms, genetic-operators
  
  > **Comments:**
  > **username_1**: What does your neural network represent? It's also not clear to me what you're trying to do. Are you trying to find the best car (that is able to drive in some environment: please, which specify which environment) by having a population of cars that you evolve my mutating? That seems to be just a **genetic algorithm**.  
  > **username_0**: my post was originall post was bigger and explained what you want but i was told i wasn''t concise enough :D  
  > **username_1**: I only asked you to ask one question, because you had multiple questions. I didn't say you should remove relevant details. Anyway, you can edit your post again.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-11-18. Score: 1 | Accepted answer
  
  The most general descriptive frameworks covering what you are trying to do are:
  
  * [Sequential decision making](https://en.wikipedia.org/wiki/Sequential_decision_making) (article is a stub, but the term a good launching point to discover different wys of modelling and solving these kind of problems)
  * [Optimal control](https://en.wikipedia.org/wiki/Optimal_control)
  
  These put some context around your problem, and might give you some pointers. For instance, reinforcement learning is an alternative approach to the evolutionary system you are trying to build.
  
  The specific AI system you appear to be building is a [genetic algorithm](https://en.wikipedia.org/wiki/Genetic_algorithm), and more specific still you are attempting to find a neural network that is optimal at a task by searching for the best network using a system of population generation, selection and mutation which repeats.
  
  There are lots of ways to set up a system like this, so your approach is not necessarily *wrong*. However, I think there are two key things that would improve what you have built so far:
  
  * Use a [fitness function](https://en.wikipedia.org/wiki/Fitness_function) for selection. Score each car, perhaps by how far it got before crashing when the episode ends. To reduce luck factor on random courses, you could make this score the mean result from e.g. 3 different courses (it is not necessary, but may address your concern that selection is too random in your case). Select some fraction of top scoring cars, or look into other selection approaches - e.g. weighted selection based on fitness score or ranking.
  * Add "sex", more properly known as genome [crossover](https://en.wikipedia.org/wiki/Crossover_(genetic_algorithm)) between selected population members. Mutating individuals is limiting because it silos improvements to a single line of ancestry - if there are two good mutations found at random you rely on that single line finding both of them. Whilst crossover allows sharing of good mutations between lines, making it much more likely that two good mutations will end up in the same individual.
  
  There is a framework called [NEAT](https://en.wikipedia.org/wiki/Neuroevolution_of_augmenting_topologies) which covers the issues above plus has other features useful for evolving neural networks. It often does well at control scenarios like the one you are considering. You may want to look into it, if your focus is mainly on solving the control problem. However, it is relatively advanced from where you are, so if your current focus to learn by building from scratch you may get more initially from implementing fitness functions and crossover yourself.
  
  > **Comments:**
  > **username_0**: thank you very much. I've seen this NEAT in the past but didn't understand it. Now that i played with my simple NN i think i understand it. (and i think that the game called "GridWorld" is doing this). My NN have a fixed topology (defined by me : 7 raycasting input, 1 speed input, 2 hidden layer of 12 and 6 neuron, 2 for speed control and direction) and i'm not encoding phenotype and therefore make crossover more difficult in my code. But i will try this from now on. i could even let the NN try different input sensor by itself (ex : angle of the raycast, tyre friction (drifing in turn), etc)  
  > **username_0**: It's ambitious but worth it i guess. i won't go anywhere if i keep playing with the same primitive NN forever and it's totally in the direction toward my end goal : a mini-game where you watch a NN learning, playing failing, evolving, ... i specifically don't want a sophisticated fitness function or error computation : it's either "win the right to reproduce/mutate" or "die without offspring". And neat seems to fit the need.  
  
  ---
  
  