Answers.Body: <p>In general, the approach of adding or altering the reward structure of a RL environment, when you are still trying to create an agent that solves the original problem, is called <a href="https://www.youtube.com/watch?v=xManAGjbx2k" rel="nofollow noreferrer">Reward Shaping</a>.</p>
  
  <p>Reward Shaping can be tricky to get right. In your case I think it may be counter-productive. The main issue is that the agent does not know the current time step as part of the state. So from its perspective, a negative reward for "timing out" actually looks like a random chance of getting that same reward in the <em>kind of states that happen later in the game</em>. The effect may be small enough chance that it doesn't really matter (after all it still gets positive rewards for hitting bricks). However, it may appear as high variance on certain game positions, making the agent learn to avoid some of them for no good reason.</p>
  
  <p>If the aim here is to try and get the agent to speed up and finish an episode, a simpler trick might be to reduce the discount factor (e.g. from <span class="math-container">$0.999$</span> to <span class="math-container">$0.99$</span>). This will cause the agent to focus on getting more short-term rewards, at the expense of long-term planning. In some environments this could be a problem of a different kind, but when the rewards are not sparse and there are not any special high-reward states that need extended setup, it should be OK.</p>
  
  <p>If the aim is to punish losing a "game life" more severely, then this is less likely to cause a problem (because the state will clearly show what to expect), although it may change what the optimal behaviour is, or how it is approached. In general the impact would be reduced risk-taking, and even though Breakout is a deterministic game, the agent is presented with a stochastic environment because each action lasts a random number of frames (2-4). With a strong penalty for losing a life, I think that the agent will be less likely to try to get high bounce angles by hitting with extreme edge of the bat. Note that you don't <em>need</em> to punish the agent for losing a life, it learns that because the episode ends and it cannot score any more points. Value functions are all about predicting future rewards, so the value of any life-losing state is always <span class="math-container">$0$</span> anyway.</p>
  
  <p>If you have more time, then there are also several extensions to basic DQN that require less sampled experience to learn an environment, or improve the resulting policies. A good source for those ideas is the paper <a href="https://arxiv.org/abs/1710.02298" rel="nofollow noreferrer">Rainbow: Combining Improvements in Deep Reinforcement Learning</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-21T20:29:09.220Z
Answers.Text: Ok, but how for example I could teach the agent that losing a life it's bad? If I don't have any negative rewards, the agend couldn't learn that, right?
Answers.UserDisplayName: JCP
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-21T20:34:50.333Z
Answers.Text: @JCP: The agent learns that because the episode ends early and it cannot score any more. There might however be less harm done if your reward shaping is just to punish losing game lives more.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-21T20:16:22.870Z
Answers.LastActivityDate: 2018-11-21T20:44:30.470Z
Answers.LastEditDate: 2018-11-21T20:44:30.470Z
Answers.UserDisplayName: Neil Slater
Body: <p>I'm trying to train a DQN, so I'm using OpenAI gym and Breakout (<a href="https://gym.openai.com/envs/Breakout-v0/" rel="nofollow noreferrer">Breakout-v0</a>).</p>
  
  <p>I have altered the reward supplied by the environment: If the episode is not completed fully, the agent gets a -10 reward. Could be this counterproductive for learning?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-11-21T14:31:45.137Z
Comments.Text: Are you adding the -10 reward yourself, or is it part of the environment as presented by OpenAI gym?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-11-21T17:59:41.363Z
Comments.Text: I'm adding it when the episode is finished and its not done
Comments.UserDisplayName: JCP
Tags: <ai-design><training><game-ai><dqn><open-ai>
Title: DQN Breakout adding an extra negative reward to help training?
UserDisplayName: JCP
ThreadText: # DQN Breakout adding an extra negative reward to help training?
  
  (Asked by: username_0 on 2018-11-21)
  
  I'm trying to train a DQN, so I'm using OpenAI gym and Breakout ([Breakout-v0](https://gym.openai.com/envs/Breakout-v0/)).
  
  I have altered the reward supplied by the environment: If the episode is not completed fully, the agent gets a -10 reward. Could be this counterproductive for learning?
  
  **Tags:** ai-design, training, game-ai, dqn, open-ai
  > **Comments:**
  > **username_1**: Are you adding the -10 reward yourself, or is it part of the environment as presented by OpenAI gym?  
  > **username_0**: I'm adding it when the episode is finished and its not done  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-11-21. Score: 2 | Accepted answer
  
  In general, the approach of adding or altering the reward structure of a RL environment, when you are still trying to create an agent that solves the original problem, is called [Reward Shaping](https://www.youtube.com/watch?v=xManAGjbx2k).
  
  Reward Shaping can be tricky to get right. In your case I think it may be counter-productive. The main issue is that the agent does not know the current time step as part of the state. So from its perspective, a negative reward for "timing out" actually looks like a random chance of getting that same reward in the *kind of states that happen later in the game*. The effect may be small enough chance that it doesn't really matter (after all it still gets positive rewards for hitting bricks). However, it may appear as high variance on certain game positions, making the agent learn to avoid some of them for no good reason.
  
  If the aim here is to try and get the agent to speed up and finish an episode, a simpler trick might be to reduce the discount factor (e.g. from $0.999$ to $0.99$). This will cause the agent to focus on getting more short-term rewards, at the expense of long-term planning. In some environments this could be a problem of a different kind, but when the rewards are not sparse and there are not any special high-reward states that need extended setup, it should be OK.
  
  If the aim is to punish losing a "game life" more severely, then this is less likely to cause a problem (because the state will clearly show what to expect), although it may change what the optimal behaviour is, or how it is approached. In general the impact would be reduced risk-taking, and even though Breakout is a deterministic game, the agent is presented with a stochastic environment because each action lasts a random number of frames (2-4). With a strong penalty for losing a life, I think that the agent will be less likely to try to get high bounce angles by hitting with extreme edge of the bat. Note that you don't *need* to punish the agent for losing a life, it learns that because the episode ends and it cannot score any more points. Value functions are all about predicting future rewards, so the value of any life-losing state is always $0$ anyway.
  
  If you have more time, then there are also several extensions to basic DQN that require less sampled experience to learn an environment, or improve the resulting policies. A good source for those ideas is the paper [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://arxiv.org/abs/1710.02298).
  
  > **Comments:**
  > **username_0**: Ok, but how for example I could teach the agent that losing a life it's bad? If I don't have any negative rewards, the agend couldn't learn that, right?  
  > **username_1**: @JCP: The agent learns that because the episode ends early and it cannot score any more. There might however be less harm done if your reward shaping is just to punish losing game lives more.  
  
  ---
  
  