Answers.Body: <p>Assume you are the snake.</p>
  
  <p>In front of you is empty. Left of you is empty. Right of you is empty. The distance to the apple is 4. The apple straight in front of you. Your length is 20.</p>
  
  <p>Can you make a good decision with this input? In which direction would you go to achieve maximum score?</p>
  
  <p>From the given input, you could go straight forward to the apple. But that might be a failure and lead to death.</p>
  
  <p>IMHO, the input state is simply not enough to make a good decision, because </p>
  
  <p>a) the snake doesn't even know in which direction it's currently moving.</p>
  
  <p>b) the snake has no idea about where its body is</p>
  
  <p>The situation could look like this:</p>
  
  <p><a href="https://i.sstatic.net/AZFmT.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/AZFmT.png" alt="Situation for the snake"></a></p>
  
  <p>The only way for the snake to move out of this trap is as indicated by the arrow, so that the tail frees the way out just in time. Your neural network does not have the necessary input to make that decision.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-21T13:36:00.203Z
Answers.LastActivityDate: 2020-01-21T13:36:00.203Z
Answers.UserDisplayName: Thomas Weller
Answers.Body: <p>There are two problems here.</p>
  
  <ol>
  <li><p>The code you posted doesn't incrimentally train your multilayer perceptron. Instead, it effectively re-randomizes the weights, and then re-fits the model each time you call .fit() at lines 35 &amp; 54. Using SKLearn's _fit() function with Incremental=true might solve this, or you can package up the data into a larger batch, and train on that offline instead after several episodes.</p></li>
  <li><p>Your reward function makes it painful to be alive, and doesn't give enough benefits through the Apples to make up for this. There are 100 squares that could contain the apple. On average, the apple will spawn about 5 squares from the snake in each direction. Since the snake can't move diagonally, that's 10 moves (5 left/right, 5 up/down). That means that if the snake plays <em>perfectly</em>, then on average, it might be able to get zero reward total. In practice, the snake will not play perfectly. This means living gives negative expected reward.</p></li>
  </ol>
  
  <p>In contrast, if the snake can kill itself, it will stop getting negative rewards. The reward function you've used is maximized by getting big enough to run into your own tail as fast as possible. The snake should be able to do this after eating 3 apples I think. There is some incentive to hunt for food well, but not much compared with hitting your own tail as soon as possible.</p>
  
  <p>If you want the snake to learn to hunt for the food, reduce the penalty for being alive to -1, or even -0.1. The snake will be much more responsive to signals from the food.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-13T07:20:08.513Z
Answers.Text: there are no walls in the game, the moment the snake goes on the edge of the map it reappears on the other end so basically it learns to never die which seems a little contradictory since he gets negative rewards for being alive
Answers.UserDisplayName: oren revenge
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-13T07:21:48.187Z
Answers.Text: @orenrevenge, you say that an episode lasts until the snake dies. How can it die? Only by hitting itself?
Answers.UserDisplayName: John Doucette
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-13T07:23:37.527Z
Answers.Text: yes thats correct
Answers.UserDisplayName: oren revenge
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-13T19:43:03.263Z
Answers.Text: @orenrevenge  This is really more of an implementation question I think, and you might have better luck on the main StackOverflow site. Looking at your code, I think your mistake is in fit_episode, on lines 35 & 53. MLPRegressor.fit() promises to produce a trained model for the data you give it, but makes no promises about  preserving the weights learned in past fits. See the SKLearn sourcecode: https://github.com/scikit-learn/scikit-learn/blob/f0ab589f/sklearn/neural_network/multilayer_perceptron.py#L604 By default, the fit is not incremental. Using _fit with incremental=true might fix this.
Answers.UserDisplayName: John Doucette
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-13T19:43:40.453Z
Answers.Text: @orenrevenge Even after you fix that though, the reward function you've picked is optimized by having the snake try to kill itself as quickly as possible. The rest of my answer still applies.
Answers.UserDisplayName: John Doucette
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-14T07:38:48.597Z
Answers.Text: that's it, the scikit-learn fit method was the problem I had to use partial_fit instead thanks a lot now I seem to get some results, the snake finally learns to go after the apple, I still need to modify the reward function for better performance according to your suggestions
Answers.UserDisplayName: oren revenge
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-13T02:13:48.507Z
Answers.LastActivityDate: 2018-09-13T19:52:13.400Z
Answers.LastEditDate: 2018-09-13T19:52:13.400Z
Answers.UserDisplayName: John Doucette
Body: <p>This is a q-learning snake using a neural network as a q function aproximator and I'm losing my mind here the current model it's worst than the initial one.</p>
  
  <p>The current model uses a 32x32x32 MLPRegressor from scikit-learn using relu as activation function and the adam solver.</p>
  
  <p>The reward function is like following:</p>
  
  <ul>
  <li>death reward = -100.0</li>
  <li>alive reward = -10.0</li>
  <li>apple reward = 100.0</li>
  </ul>
  
  <p>The features extracted from each state are the following:</p>
  
  <ol>
  <li>what is in front of the snake's head(apple, empty, snake)</li>
  <li>what is in the left of the snake's head</li>
  <li>what is in the right of the snake's head</li>
  <li>euclidian distance between head and apple</li>
  <li>the direction from head to the apple measured in radians</li>
  <li>length of the snake</li>
  </ol>
  
  <p>One episode consists of the snake playing until it dies, I'm also using in training a probability epsilon that represent the probability that the snake will take a random action if this isn't satisfied the snake will take the action for which the neural network gives the biggest score, this epsilon probability gradually decrements after each iteration.</p>
  
  <p>The episode is learned by the regressor in reverse order one statet-action at a time.</p>
  
  <p>However the neural network fails too aproximate the q function, no matter how many iterations the snake takes the same action for any state.</p>
  
  <p>Things I tried:</p>
  
  <ul>
  <li>changing the structure of the neural network</li>
  <li>changing the reward function</li>
  <li>changing the features extracted, I even tried passing the whole map to the network</li>
  </ul>
  
  <p>Code (python): <a href="https://pastebin.com/57qLbjQZ" rel="nofollow noreferrer">https://pastebin.com/57qLbjQZ</a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-12T21:10:18.057Z
Comments.Text: Can you provide a snippet of your code.
Comments.UserDisplayName: quintumnia
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-12T21:18:18.053Z
Comments.Text: the pastebin link to the code its in the post
Comments.UserDisplayName: oren revenge
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-09-13T19:44:12.887Z
Comments.Text: @orenrevenge I've updated my answer to address the code you posted too.
Comments.UserDisplayName: John Doucette
Tags: <reinforcement-learning>
Title: Snake game: snake converges to going in the same direction every time
UserDisplayName: oren revenge
ThreadText: # Snake game: snake converges to going in the same direction every time
  
  (Asked by: username_0 on 2018-09-12)
  
  This is a q-learning snake using a neural network as a q function aproximator and I'm losing my mind here the current model it's worst than the initial one.
  
  The current model uses a 32x32x32 MLPRegressor from scikit-learn using relu as activation function and the adam solver.
  
  The reward function is like following:
  
  * death reward = -100.0
  * alive reward = -10.0
  * apple reward = 100.0
  
  The features extracted from each state are the following:
  
  1. what is in front of the snake's head(apple, empty, snake)
  2. what is in the left of the snake's head
  3. what is in the right of the snake's head
  4. euclidian distance between head and apple
  5. the direction from head to the apple measured in radians
  6. length of the snake
  
  One episode consists of the snake playing until it dies, I'm also using in training a probability epsilon that represent the probability that the snake will take a random action if this isn't satisfied the snake will take the action for which the neural network gives the biggest score, this epsilon probability gradually decrements after each iteration.
  
  The episode is learned by the regressor in reverse order one statet-action at a time.
  
  However the neural network fails too aproximate the q function, no matter how many iterations the snake takes the same action for any state.
  
  Things I tried:
  
  * changing the structure of the neural network
  * changing the reward function
  * changing the features extracted, I even tried passing the whole map to the network
  
  Code (python): <https://pastebin.com/57qLbjQZ>
  
  **Tags:** reinforcement-learning
  > **Comments:**
  > **username_1**: Can you provide a snippet of your code.  
  > **username_0**: the pastebin link to the code its in the post  
  > **username_2**: @orenrevenge I've updated my answer to address the code you posted too.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2018-09-13. Score: 4 | Accepted answer
  
  There are two problems here.
  
  1. The code you posted doesn't incrimentally train your multilayer perceptron. Instead, it effectively re-randomizes the weights, and then re-fits the model each time you call .fit() at lines 35 & 54. Using SKLearn's \_fit() function with Incremental=true might solve this, or you can package up the data into a larger batch, and train on that offline instead after several episodes.
  2. Your reward function makes it painful to be alive, and doesn't give enough benefits through the Apples to make up for this. There are 100 squares that could contain the apple. On average, the apple will spawn about 5 squares from the snake in each direction. Since the snake can't move diagonally, that's 10 moves (5 left/right, 5 up/down). That means that if the snake plays *perfectly*, then on average, it might be able to get zero reward total. In practice, the snake will not play perfectly. This means living gives negative expected reward.
  
  In contrast, if the snake can kill itself, it will stop getting negative rewards. The reward function you've used is maximized by getting big enough to run into your own tail as fast as possible. The snake should be able to do this after eating 3 apples I think. There is some incentive to hunt for food well, but not much compared with hitting your own tail as soon as possible.
  
  If you want the snake to learn to hunt for the food, reduce the penalty for being alive to -1, or even -0.1. The snake will be much more responsive to signals from the food.
  
  > **Comments:**
  > **username_0**: there are no walls in the game, the moment the snake goes on the edge of the map it reappears on the other end so basically it learns to never die which seems a little contradictory since he gets negative rewards for being alive  
  > **username_2**: @orenrevenge, you say that an episode lasts until the snake dies. How can it die? Only by hitting itself?  
  > **username_0**: yes thats correct  
  > **username_2**: @orenrevenge  This is really more of an implementation question I think, and you might have better luck on the main StackOverflow site. Looking at your code, I think your mistake is in fit_episode, on lines 35 & 53. MLPRegressor.fit() promises to produce a trained model for the data you give it, but makes no promises about  preserving the weights learned in past fits. See the SKLearn sourcecode: https://github.com/scikit-learn/scikit-learn/blob/f0ab589f/sklearn/neural_network/multilayer_perceptron.py#L604 By default, the fit is not incremental. Using _fit with incremental=true might fix this.  
  > **username_2**: @orenrevenge Even after you fix that though, the reward function you've picked is optimized by having the snake try to kill itself as quickly as possible. The rest of my answer still applies.  
  > **username_0**: that's it, the scikit-learn fit method was the problem I had to use partial_fit instead thanks a lot now I seem to get some results, the snake finally learns to go after the apple, I still need to modify the reward function for better performance according to your suggestions  
  
  ---
  
  ### Answer by username_3 on 2020-01-21. Score: 2
  
  Assume you are the snake.
  
  In front of you is empty. Left of you is empty. Right of you is empty. The distance to the apple is 4. The apple straight in front of you. Your length is 20.
  
  Can you make a good decision with this input? In which direction would you go to achieve maximum score?
  
  From the given input, you could go straight forward to the apple. But that might be a failure and lead to death.
  
  IMHO, the input state is simply not enough to make a good decision, because
  
  a) the snake doesn't even know in which direction it's currently moving.
  
  b) the snake has no idea about where its body is
  
  The situation could look like this:
  
  [![Situation for the snake](https://i.sstatic.net/AZFmT.png)](https://i.sstatic.net/AZFmT.png)
  
  The only way for the snake to move out of this trap is as indicated by the arrow, so that the tail frees the way out just in time. Your neural network does not have the necessary input to make that decision.
  
  
  ---
  
  