Answers.Body: <p>There might be some cases, say, once the agent clearing a brick obtaining a positive reward and then forever bouncing the ball on the same side of the screen would have a higher cumulative reward than trying to clear as many bricks as possible but then got many negative rewards which might be significant compared to positive reward. It's like a conservative game player, once winning a game initially then stops playing completely. Or it's like the case if getting a downvote of an answer is equal to getting an upvote in terms of scores, then many more people would be reluctant to answer any question.</p>
  <p>Therefore appropriate reward engineering is crucial. In certain scenarios a poorly designed reward function might inadvertently incentivize the agent to adopt <em>suboptimal</em> strategies that maximize cumulative rewards without truly achieving the main objectives of the task.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-26T11:14:14.577Z
Answers.Text: Worth noting that the -ve reward for missing the ball with the paddle is far more immediate and obvious response from the game environment, when considering relationship between action and reward, than the +ve reward from hitting a brick which is significantly delayed from the action that caused the ball to go in the right direction
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-25T20:09:58.247Z
Answers.LastActivityDate: 2024-02-25T20:09:58.247Z
Answers.UserDisplayName: cinch
Body: <p>The following is from page 17 of  &quot;Michael Hu, “The Art of Reinforcement Learning: Fundamentals, Mathematics, and Implementations with Python”, Apress, 2023&quot;</p>
  <p><a href="https://link.springer.com/book/10.1007/978-1-4842-9606-6" rel="nofollow noreferrer">https://link.springer.com/book/10.1007/978-1-4842-9606-6</a></p>
  <blockquote>
  <p>An example of good reward engineering is in the game of Atari Breakout, where the goal of the agent is to clear all the bricks at the top of the screen by bouncing a ball off a paddle. One way to design a reward function for this game is to give the agent a positive reward for each brick it clears and a negative reward for each time the ball passes the paddle and goes out of bounds. However, this reward function alone may not lead to optimal behavior, as the agent may learn to exploit a loophole by simply bouncing the ball back and forth on the same side of the screen without actually clearing any bricks.</p>
  </blockquote>
  <p>This part is not clear:</p>
  <blockquote>
  <p>However, this reward function alone may not lead to optimal behavior, as the agent may learn to exploit a loophole by simply bouncing the ball back and forth on the same side of the screen without actually clearing any bricks.</p>
  </blockquote>
  <p>Why should the agent bounce the ball back and forth on the same side of the screen? Is there a reward in this case?</p>
  
Tags: <reinforcement-learning><reward-design><atari-games>
Title: Why should the agent bounce the ball back and forth on the same side of the screen in Atari Breakout?
UserDisplayName: DSPinfinity
ThreadText: # Why should the agent bounce the ball back and forth on the same side of the screen in Atari Breakout?
  
  (Asked by: username_0 on 2024-02-25)
  
  The following is from page 17 of "Michael Hu, “The Art of Reinforcement Learning: Fundamentals, Mathematics, and Implementations with Python”, Apress, 2023"
  
  <https://link.springer.com/book/10.1007/978-1-4842-9606-6>
  
  > An example of good reward engineering is in the game of Atari Breakout, where the goal of the agent is to clear all the bricks at the top of the screen by bouncing a ball off a paddle. One way to design a reward function for this game is to give the agent a positive reward for each brick it clears and a negative reward for each time the ball passes the paddle and goes out of bounds. However, this reward function alone may not lead to optimal behavior, as the agent may learn to exploit a loophole by simply bouncing the ball back and forth on the same side of the screen without actually clearing any bricks.
  
  This part is not clear:
  
  > However, this reward function alone may not lead to optimal behavior, as the agent may learn to exploit a loophole by simply bouncing the ball back and forth on the same side of the screen without actually clearing any bricks.
  
  Why should the agent bounce the ball back and forth on the same side of the screen? Is there a reward in this case?
  
  **Tags:** reinforcement-learning, reward-design, atari-games
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-02-25. Score: 2 | Accepted answer
  
  There might be some cases, say, once the agent clearing a brick obtaining a positive reward and then forever bouncing the ball on the same side of the screen would have a higher cumulative reward than trying to clear as many bricks as possible but then got many negative rewards which might be significant compared to positive reward. It's like a conservative game player, once winning a game initially then stops playing completely. Or it's like the case if getting a downvote of an answer is equal to getting an upvote in terms of scores, then many more people would be reluctant to answer any question.
  
  Therefore appropriate reward engineering is crucial. In certain scenarios a poorly designed reward function might inadvertently incentivize the agent to adopt *suboptimal* strategies that maximize cumulative rewards without truly achieving the main objectives of the task.
  
  > **Comments:**
  > **username_2**: Worth noting that the -ve reward for missing the ball with the paddle is far more immediate and obvious response from the game environment, when considering relationship between action and reward, than the +ve reward from hitting a brick which is significantly delayed from the action that caused the ball to go in the right direction  
  
  ---
  
  