Answers.Body: <p>Your network finds the infinite loop and notices that this has the best reward (0). This is probably because it hasn't found a path to eating the apple (through exploration).</p>
  
  <p>Reducing the discount factor will only make the long term rewards less valuable. So it will learn eating the apple even slower.</p>
  
  <p>I don't know what you are using as inputs for your network, but maybe changing your reward system could help. For example, you could give your network a reward if it advances in the direction of the apple. This way, your network will be encouraged to find the apple more than it is now.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-09T14:14:01.537Z
Answers.Text: I'm using a convolutional neural network so I don't think that approach would be easily implemened I think.
Answers.UserDisplayName: KaneM
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-09T14:15:48.643Z
Answers.Text: My thinking was that reducing the discount factor would incentivise the snake to get the reward sooner rather than later because if it waits a while to get it then it will recieve less reward
Answers.UserDisplayName: KaneM
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-09T19:42:23.613Z
Answers.Text: How is this not easily implemented? After each decision you measure distance to the apple, if it is smaller than last distance you get reward +1.
Answers.UserDisplayName: Lustwelpintje
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-10T10:54:08.570Z
Answers.Text: You are right. I feel that may cause some unwanted behavior though. It might keep going back and forth between the apple and not actually getting it as it can keep getting reward this way. I will try it out and see if it works
Answers.UserDisplayName: KaneM
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-10T11:18:17.840Z
Answers.Text: This is of course the problem with reward shaping - spare rewards. Try this out, using different rewards for moving away form the apple, towards the apple, eating the apple, and dying. you can even give a negative reward for not eating the apple within a certain time period
Answers.UserDisplayName: Lustwelpintje
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-16T07:42:48.830Z
Answers.Text: Another approach is that you should give negative rewards for wandering too long. also during training, help your agent to experince wandering by not allowing to randomize exploration for small period of time. this way it helps with unnecessary detours.
Answers.UserDisplayName: Ashkan Ghodrat
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-09T13:13:18.027Z
Answers.LastActivityDate: 2020-03-09T13:13:18.027Z
Answers.UserDisplayName: Lustwelpintje
Body: <p>I have made a Deep Q Network for the game snake but unfortunately, the snake exhibits some unwanted behavior. It generally does quite well but sometimes it gets stuck in an infinite loop that it can't escape and at the start of the game it takes a very long route to the apple rather than taking a more direct route.</p>
  
  <p>The discount factor per time step is 0.99. The Snake gets a reward of +9 for getting an Apple and -1 for dying. Does anybody have any recommendations on how I should tune the hyperparameters/reward function to minimize this unwanted behavior?</p>
  
  <p>I was thinking that reducing the discount factor may be a good idea?</p>
  
Tags: <reinforcement-learning><rewards>
Title: How to incentivise snake to go straight to apple?
UserDisplayName: KaneM
ThreadText: # How to incentivise snake to go straight to apple?
  
  (Asked by: username_0 on 2020-03-09)
  
  I have made a Deep Q Network for the game snake but unfortunately, the snake exhibits some unwanted behavior. It generally does quite well but sometimes it gets stuck in an infinite loop that it can't escape and at the start of the game it takes a very long route to the apple rather than taking a more direct route.
  
  The discount factor per time step is 0.99. The Snake gets a reward of +9 for getting an Apple and -1 for dying. Does anybody have any recommendations on how I should tune the hyperparameters/reward function to minimize this unwanted behavior?
  
  I was thinking that reducing the discount factor may be a good idea?
  
  **Tags:** reinforcement-learning, rewards
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-03-09. Score: 1
  
  Your network finds the infinite loop and notices that this has the best reward (0). This is probably because it hasn't found a path to eating the apple (through exploration).
  
  Reducing the discount factor will only make the long term rewards less valuable. So it will learn eating the apple even slower.
  
  I don't know what you are using as inputs for your network, but maybe changing your reward system could help. For example, you could give your network a reward if it advances in the direction of the apple. This way, your network will be encouraged to find the apple more than it is now.
  
  > **Comments:**
  > **username_0**: I'm using a convolutional neural network so I don't think that approach would be easily implemened I think.  
  > **username_0**: My thinking was that reducing the discount factor would incentivise the snake to get the reward sooner rather than later because if it waits a while to get it then it will recieve less reward  
  > **username_1**: How is this not easily implemented? After each decision you measure distance to the apple, if it is smaller than last distance you get reward +1.  
  > **username_0**: You are right. I feel that may cause some unwanted behavior though. It might keep going back and forth between the apple and not actually getting it as it can keep getting reward this way. I will try it out and see if it works  
  > **username_1**: This is of course the problem with reward shaping - spare rewards. Try this out, using different rewards for moving away form the apple, towards the apple, eating the apple, and dying. you can even give a negative reward for not eating the apple within a certain time period  
  > **username_2**: Another approach is that you should give negative rewards for wandering too long. also during training, help your agent to experince wandering by not allowing to randomize exploration for small period of time. this way it helps with unnecessary detours.  
  
  ---
  
  