Answers.Body: <p>I would suggest looking at various optimisations used by Deep Mind when they originally developed DQN. You can check <a href="https://paperswithcode.com/method/rainbow-dqn" rel="nofollow noreferrer">their &quot;rainbow&quot; paper</a> for ideas that improved performance.</p>
  <p>In addition, I think you are missing the idea of frame stacking, which should help the agent better understand <em>velocity</em> which is currently not in your state representation, although the position of the head will often be a clue. To frame stack, change the representation to include the last few  turns (3 turns total might be enough for you) and concatenate the channels. You could also take the opportunity to flatten the representation of a single frame*, since each grid point only ever contains a single entity (e.g. you could use 0 for empty, 1 for apple, -1 for snake body and -0.5 for snake head). That should reduce the number of parameters required for the estimator, which in turn may improve model performance.</p>
  <p>Reward values like +100 might be a problem in that they could cause large gradients in a neural network whilst learning them. It is worth checking to see what happens when you normalise your reward values to just +1 for eating an apple. Termination of an episode when the snake crashes into itself, and largish discount factor, <span class="math-container">$\gamma = 0.99$</span> should be enough to discourage the snake from hitting itself or walls.</p>
  <hr />
  <p>* In DQN they turned the game inputs into greyscale, but if you do that with your current colour choices, all the entities would have the same value. However using greyscale could work for you as well, if you had all the entities display with different colour intensities.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-13T20:09:05.213Z
Answers.LastActivityDate: 2022-12-13T20:15:54.013Z
Answers.LastEditDate: 2022-12-13T20:15:54.013Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>Here are your options:</p>
  <ol>
  <li>Try different starting points in the path.</li>
  <li>Try creating more paths to train your snake.</li>
  <li>Try using genetic algorithms to &quot;evolve&quot; your algorithm more systematically.</li>
  <li>I don't think that snakes eat apples, so try changing the reward to a mouse if you are serious about getting good performance out of your snake.</li>
  <li>Some snakes are smarter than others, so try using a cobras, pythons, etc.  Do not use titanaboas because they are extinct, probably for a reason.</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-13T17:49:35.723Z
Answers.LastActivityDate: 2022-11-13T17:49:35.723Z
Answers.UserDisplayName: Snehal Patel
Body: <p>I'm training a Deep Q-learning model on a snake game and I would like some ideas on how to improve the model and maybe also efficiency of training it.</p>
  <p>The game is currently set to a 12x12 grid, a blue snake with a green head and a red apple. The network is fed with 3x12x12 input parameters (RGB, width, height) and gets a positive reward when an apple is eaten and a negative reward when it collides with something.</p>
  <p>It does learn, but plateaus around 12-13 apples per round (on average) after 3 million steps:
  <a href="https://i.sstatic.net/rzS1g.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/rzS1g.png" alt="enter image description here" /></a></p>
  <p>What I have tried:
  Giving a partly reward on the steps before a &quot;real&quot; reward. For example:</p>
  <pre><code>Step    Action        Reward
  N       Go straight      100
  N-1     Go straight       50
  N-2     Go straight       33
  N-3     Go left           25
  </code></pre>
  <p>That was just an idea but it does not seem to work as I hoped.</p>
  <p>What else can I try? What I don't want to do is tinker with the game, I just want the visual input and nothing else.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-16T17:51:03.017Z
Comments.Text: Hi @DanneP and welcome to AI Stack Exchange! If possible, could you supply a few more details, such as the specific variant of the DQN algorithm you are using? That would help us determine what improvements could be made. Thanks!
Comments.UserDisplayName: DeepQZero
Tags: <reinforcement-learning><convolutional-neural-networks><dqn>
Title: Ways to improve DQN model learning snake?
UserDisplayName: DanneP
ThreadText: # Ways to improve DQN model learning snake?
  
  (Asked by: username_0 on 2022-11-13)
  
  I'm training a Deep Q-learning model on a snake game and I would like some ideas on how to improve the model and maybe also efficiency of training it.
  
  The game is currently set to a 12x12 grid, a blue snake with a green head and a red apple. The network is fed with 3x12x12 input parameters (RGB, width, height) and gets a positive reward when an apple is eaten and a negative reward when it collides with something.
  
  It does learn, but plateaus around 12-13 apples per round (on average) after 3 million steps:
  [![enter image description here](https://i.sstatic.net/rzS1g.png)](https://i.sstatic.net/rzS1g.png)
  
  What I have tried:
  Giving a partly reward on the steps before a "real" reward. For example:
  
  ```
  Step    Action        Reward
  N       Go straight      100
  N-1     Go straight       50
  N-2     Go straight       33
  N-3     Go left           25
  
  ```
  
  That was just an idea but it does not seem to work as I hoped.
  
  What else can I try? What I don't want to do is tinker with the game, I just want the visual input and nothing else.
  
  **Tags:** reinforcement-learning, convolutional-neural-networks, dqn
  
  > **Comments:**
  > **username_1**: Hi @username_0 and welcome to AI Stack Exchange! If possible, could you supply a few more details, such as the specific variant of the DQN algorithm you are using? That would help us determine what improvements could be made. Thanks!  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-12-13. Score: 1
  
  I would suggest looking at various optimisations used by Deep Mind when they originally developed DQN. You can check [their "rainbow" paper](https://paperswithcode.com/method/rainbow-dqn) for ideas that improved performance.
  
  In addition, I think you are missing the idea of frame stacking, which should help the agent better understand *velocity* which is currently not in your state representation, although the position of the head will often be a clue. To frame stack, change the representation to include the last few turns (3 turns total might be enough for you) and concatenate the channels. You could also take the opportunity to flatten the representation of a single frame\*, since each grid point only ever contains a single entity (e.g. you could use 0 for empty, 1 for apple, -1 for snake body and -0.5 for snake head). That should reduce the number of parameters required for the estimator, which in turn may improve model performance.
  
  Reward values like +100 might be a problem in that they could cause large gradients in a neural network whilst learning them. It is worth checking to see what happens when you normalise your reward values to just +1 for eating an apple. Termination of an episode when the snake crashes into itself, and largish discount factor, $\gamma = 0.99$ should be enough to discourage the snake from hitting itself or walls.
  
  ---
  
  \* In DQN they turned the game inputs into greyscale, but if you do that with your current colour choices, all the entities would have the same value. However using greyscale could work for you as well, if you had all the entities display with different colour intensities.
  
  
  ---
  
  ### Answer by username_3 on 2022-11-13. Score: 1
  
  Here are your options:
  
  1. Try different starting points in the path.
  2. Try creating more paths to train your snake.
  3. Try using genetic algorithms to "evolve" your algorithm more systematically.
  4. I don't think that snakes eat apples, so try changing the reward to a mouse if you are serious about getting good performance out of your snake.
  5. Some snakes are smarter than others, so try using a cobras, pythons, etc. Do not use titanaboas because they are extinct, probably for a reason.
  
  
  ---
  
  