Answers.Body: <p>I think the main issue here is that you are trying to train the <code>snake</code> (network) on images. This will create a lot of issues a there are no set parameters that the model can learn from.</p>
  
  <p>From images, there is no logical way to define the boundary, directions and objects on the board. It will be much easier to write a simple computer vision script of game API to provide actual meaningful inputs to the model.</p>
  
  <p><a href="https://towardsdatascience.com/today-im-going-to-talk-about-a-small-practical-example-of-using-neural-networks-training-one-to-6b2cbd6efdb3" rel="nofollow noreferrer">Here</a> is a great article on building a model to play the snake game. The author also provides the game API for input along with example code to train the snake game.</p>
  
  <p><strong>Final results from the model</strong></p>
  
  <p><a href="https://i.sstatic.net/du7ZI.gif" rel="nofollow noreferrer"><img src="https://i.sstatic.net/du7ZI.gif" alt="Snake agent results"></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-22T19:04:13.783Z
Answers.Text: Thanks for you answer.
  The problem is that I need to do it in that way for academical pouposes
  I'm taking inspiration from https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf where the state is only the image and the reward is where I have more freedom.
  
  This is another example https://medium.com/@hugo.sjoberg88/using-reinforcement-learning-and-q-learning-to-play-snake-28423dd49e9b
Answers.UserDisplayName: Roberto Aureli
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-18T11:34:18.673Z
Answers.Text: Is snake game harder for dqn than any atari game google use to train their dqn?
Answers.UserDisplayName: JustOneMan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-22T18:56:11.383Z
Answers.LastActivityDate: 2019-06-22T18:56:11.383Z
Answers.UserDisplayName: skillsmuggler
Body: <p>I'm using a DQN Algorithm to play Snake.</p>
  
  <p>The input of the neural network is a stack of 4 images taken from the games 80x80.</p>
  
  <p>The output is an array of 4 values, one for every direction.</p>
  
  <p>The problem is that the program does not converge and I've a lot of doubts in the replay function, where I train the neural network over a batch of 32 events.</p>
  
  <p>That's the snippet:</p>
  
  
  
  <pre><code>def replay(self, batch_size):
  
      minibatch = random.sample(self.memory, batch_size)
  
      for state, action, reward, next_state, done in minibatch:
  
          target = reward
  
          if not done:
              target = (reward + self.gamma *
                        np.amax(self.model.predict(next_state)[0]))
          target_f = self.model.predict(state)
          target_f[0][action] = target
          self.model.fit(state, target_f, epochs=1, verbose=0)
  
      if self.epsilon &gt; self.epsilon_min:
          self.epsilon *= self.epsilon_decay`
  </code></pre>
  
  <p>Targets are:</p>
  
  <ul>
  <li><strong>+1</strong> for eating an apple</li>
  <li><strong>0</strong> for doing a movement without dying</li>
  <li><strong>-1000</strong> for hitting a wall or the snake hitting himself</li>
  </ul>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-22T18:10:22.997Z
Comments.Text: Hi! questions about implementation are off-topic here, you may want to try other SE. Anyway, isn't -1000 a bit too much?
Comments.UserDisplayName: olinarr
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-22T18:15:08.917Z
Comments.Text: Sorry.
  I was trying to convince the snake to not die but I can't get around the Q approximation with the neural network
Comments.UserDisplayName: Roberto Aureli
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-10T19:31:45.017Z
Comments.Text: @RobertoAureli Did you make some progress? I have the same problem that the snake is not converging
Comments.UserDisplayName: greedsin
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-10T20:31:51.783Z
Comments.Text: @greedsin Unfortunately not, I stopped the project
Comments.UserDisplayName: Roberto Aureli
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-10T22:06:29.113Z
Comments.Text: @greedsin -1000 is a really bad scaling for the loss. it will explode the gradient and not lead to good results. you should give something like +1/10 for eat and -1 for die. also deepq learning is notorious hard on pixels. you might want to look into Data regularized Q Learning.
Comments.UserDisplayName: tnfru
Tags: <reinforcement-learning><python><tensorflow><q-learning><dqn>
Title: Problem over DQN Algorithm not converging on snake
UserDisplayName: Roberto Aureli
ThreadText: # Problem over DQN Algorithm not converging on snake
  
  (Asked by: username_0 on 2019-06-22)
  
  I'm using a DQN Algorithm to play Snake.
  The input of the neural network is a stack of 4 images taken from the games 80x80.
  The output is an array of 4 values, one for every direction.
  The problem is that the program does not converge and I've a lot of doubts in the replay function, where I train the neural network over a batch of 32 events.
  That's the snippet:
  def replay(self, batch_size):
  
      minibatch = random.sample(self.memory, batch_size)
  
      for state, action, reward, next_state, done in minibatch:
  
          target = reward
  
          if not done:
              target = (reward + self.gamma *
                        np.amax(self.model.predict(next_state)[0]))
          target_f = self.model.predict(state)
          target_f[0][action] = target
          self.model.fit(state, target_f, epochs=1, verbose=0)
  
      if self.epsilon > self.epsilon_min:
          self.epsilon *= self.epsilon_decay`
  
  Targets are:
  
  +1 for eating an apple
  0 for doing a movement without dying
  -1000 for hitting a wall or the snake hitting himself
  
  **Tags:** reinforcement-learning, python, tensorflow, q-learning, dqn
  > **Comments:**
  > **username_1**: Hi! questions about implementation are off-topic here, you may want to try other SE. Anyway, isn't -1000 a bit too much?  
  > **username_0**: Sorry. I was trying to convince the snake to not die but I can't get around the Q approximation with the neural network  
  > **username_2**: @RobertoAureli Did you make some progress? I have the same problem that the snake is not converging  
  > **username_0**: @greedsin Unfortunately not, I stopped the project  
  > **username_3**: @greedsin -1000 is a really bad scaling for the loss. it will explode the gradient and not lead to good results. you should give something like +1/10 for eat and -1 for die. also deepq learning is notorious hard on pixels. you might want to look into Data regularized Q Learning.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_4 on 2019-06-22. Score: 0
  
  I think the main issue here is that you are trying to train the `snake` (network) on images. This will create a lot of issues a there are no set parameters that the model can learn from.
  
  From images, there is no logical way to define the boundary, directions and objects on the board. It will be much easier to write a simple computer vision script of game API to provide actual meaningful inputs to the model.
  
  [Here](https://towardsdatascience.com/today-im-going-to-talk-about-a-small-practical-example-of-using-neural-networks-training-one-to-6b2cbd6efdb3) is a great article on building a model to play the snake game. The author also provides the game API for input along with example code to train the snake game.
  
  **Final results from the model**
  
  [![Snake agent results](https://i.sstatic.net/du7ZI.gif)](https://i.sstatic.net/du7ZI.gif)
  
  > **Comments:**
  > **username_0**: Thanks for you answer. The problem is that I need to do it in that way for academical pouposes I'm taking inspiration from https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf where the state is only the image and the reward is where I have more freedom.  This is another example https://medium.com/@hugo.sjoberg88/using-reinforcement-learning-and-q-learning-to-play-snake-28423dd49e9b  
  > **username_5**: Is snake game harder for dqn than any atari game google use to train their dqn?  
  
  ---
  
  