Answers.Body: <p><a href="http://incompleteideas.net/book/the-book-2nd.html" rel="nofollow noreferrer">Sutton and Barto</a> state, "The reward signal is your way of communicating to the robot [agent] <em>what</em> you want it to achieve, not <em>how</em> you want it achieved." Since you stated that the goal is to reach the finish line first, then a reward of <span class="math-container">$1$</span> for winning, <span class="math-container">$0$</span> for losing, and <span class="math-container">$0$</span> at all other time steps seems to fit that narrative. If a draw is identical to a loss, then it should provide reward <span class="math-container">$0$</span>; otherwise, a reward of <span class="math-container">$0.5$</span> seems reasonable. These rewards provide <em>model interpretability</em>: an expected return of <span class="math-container">$p$</span> (estimated with a state-value or action-value) at a certain state under the current policy would signify a <span class="math-container">$p$</span> chance of winning. Also, keeping the rewards in absolute value at most 1 can aid in training speed and prevent divergence, but it often isn't necessary for deep reinforcement learning problems. You most certainly can add other rewards based on partial progress towards the goal, but as it seems you found out, they may lead to incorrect results.</p>
  
  <p>That being said, I would focus on the <strong>training process instead of a finely-tuned reward signal</strong>. Since there is a known goal state in the racing game (the finish line), I suggest training the RL agent by first initializing all racer agents only a few steps away from the goal state at the beginning of each episode. These episodes are shorter and therefore should provide a more dense reward signal. When your RL agent has learned a winning policy (e.g. wins more often than not), then initialize the agents slightly further from the goal state at the beginning of each episode. Also, <strong>continue to use and train the same neural network</strong>. Since the neural network presumably knows a winning policy at states near the goal state, then by initializing the agents only a few states further back, the RL agent is given an warm start and only needs to learn a policy for a few more states. The policy encoded by the neural network essentially contains a refined reward signal for the states close to the goal state since it is based on a winning policy; this helps prevent the sparsity problem caused by only supplying a reward at episode completion. You can repeat this process by initializing the agents slightly further from the goal state once the RL agent has learned a winning policy while continuing to use and train the same neural network. </p>
  
  <p>Depending on your access to the environment internals, you may need other analogous approaches. For example, you could initialize the agents at the original starting line (i.e. not partway down the map) and then see which agent makes it <span class="math-container">$n$</span> units down the map first to determine the winner. Once a winning policy is learned by the RL agent, then gradually increase <span class="math-container">$n$</span> until <span class="math-container">$n$</span> matches the distance from the starting line to the finish line. Since it seems like you have <code>distance traveled</code> and <code>distance to the opponent</code> as features, you may instead try this method if you are unable to initialize the agents wherever you want on the map and instead can only initialize them on the starting line.</p>
  
  <p>A notable benefit of the overall approach is that you can more quickly debug your algorithm on the easier environments (i.e. ones with shorter episode lengths) to be confident that the learning process is correct and focus your efforts elsewhere (e.g. the training process, including the reward signal).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-11T21:52:28.267Z
Answers.LastActivityDate: 2020-06-11T21:52:28.267Z
Answers.UserDisplayName: DeepQZero
Body: <p>I'm building an agent for a racing game. In this game, there is a randomized map where there are speed boosts for the player to pick up and obstacles that act to slow the player down. The goal of the game is to reach the finishing line before the opponent.</p>
  <p>While working on this problem, I've realized that we can almost forget about the presence of our opponent and just focus on getting the agent to the finish line as quickly as possible.</p>
  <p>I started with a simple</p>
  <ul>
  <li><span class="math-container">$-1$</span> reward for every timestep</li>
  <li><span class="math-container">$+100$</span> reward for winning, and</li>
  <li><span class="math-container">$-100$</span> for losing.</li>
  </ul>
  <p>When I was experimenting with this, I felt like the rewards may be too <em>sparse</em>, as my agent was converging to pretty poor average returns. I iterated to a function of speed and distance travelled (along with the <span class="math-container">$+100$</span> reward), but, after some experimentation, I started feeling like the agent might be able to achieve high returns without necessarily being the fastest to the finish line.</p>
  <p>I'm thinking that I return to the first approach and possibly add in some reward for being in the first place (as a function of the opponent's distance behind the agent).</p>
  <p><em>What else could I try? Should I try and spread the positive rewards out more for good behavior? should I create additional rewards/penalties for perhaps hitting obstacles and using boosts or can I expect the agent to learn the correlation?</em></p>
  
Tags: <reinforcement-learning><game-ai><rewards><reward-design><reward-shaping>
Title: How should I design the reward function for racing game (where the goal is to reach finishing line before the opponent)?
UserDisplayName: Ross Kohler
ThreadText: # How should I design the reward function for racing game (where the goal is to reach finishing line before the opponent)?
  
  (Asked by: username_0 on 2020-05-01)
  
  I'm building an agent for a racing game. In this game, there is a randomized map where there are speed boosts for the player to pick up and obstacles that act to slow the player down. The goal of the game is to reach the finishing line before the opponent.
  
  While working on this problem, I've realized that we can almost forget about the presence of our opponent and just focus on getting the agent to the finish line as quickly as possible.
  
  I started with a simple
  
  * $-1$ reward for every timestep
  * $+100$ reward for winning, and
  * $-100$ for losing.
  
  When I was experimenting with this, I felt like the rewards may be too *sparse*, as my agent was converging to pretty poor average returns. I iterated to a function of speed and distance travelled (along with the $+100$ reward), but, after some experimentation, I started feeling like the agent might be able to achieve high returns without necessarily being the fastest to the finish line.
  
  I'm thinking that I return to the first approach and possibly add in some reward for being in the first place (as a function of the opponent's distance behind the agent).
  
  *What else could I try? Should I try and spread the positive rewards out more for good behavior? should I create additional rewards/penalties for perhaps hitting obstacles and using boosts or can I expect the agent to learn the correlation?*
  
  **Tags:** reinforcement-learning, game-ai, rewards, reward-design, reward-shaping
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-11. Score: 2
  
  [Sutton and Barto](http://incompleteideas.net/book/the-book-2nd.html) state, "The reward signal is your way of communicating to the robot [agent] *what* you want it to achieve, not *how* you want it achieved." Since you stated that the goal is to reach the finish line first, then a reward of $1$ for winning, $0$ for losing, and $0$ at all other time steps seems to fit that narrative. If a draw is identical to a loss, then it should provide reward $0$; otherwise, a reward of $0.5$ seems reasonable. These rewards provide *model interpretability*: an expected return of $p$ (estimated with a state-value or action-value) at a certain state under the current policy would signify a $p$ chance of winning. Also, keeping the rewards in absolute value at most 1 can aid in training speed and prevent divergence, but it often isn't necessary for deep reinforcement learning problems. You most certainly can add other rewards based on partial progress towards the goal, but as it seems you found out, they may lead to incorrect results.
  
  That being said, I would focus on the **training process instead of a finely-tuned reward signal**. Since there is a known goal state in the racing game (the finish line), I suggest training the RL agent by first initializing all racer agents only a few steps away from the goal state at the beginning of each episode. These episodes are shorter and therefore should provide a more dense reward signal. When your RL agent has learned a winning policy (e.g. wins more often than not), then initialize the agents slightly further from the goal state at the beginning of each episode. Also, **continue to use and train the same neural network**. Since the neural network presumably knows a winning policy at states near the goal state, then by initializing the agents only a few states further back, the RL agent is given an warm start and only needs to learn a policy for a few more states. The policy encoded by the neural network essentially contains a refined reward signal for the states close to the goal state since it is based on a winning policy; this helps prevent the sparsity problem caused by only supplying a reward at episode completion. You can repeat this process by initializing the agents slightly further from the goal state once the RL agent has learned a winning policy while continuing to use and train the same neural network.
  
  Depending on your access to the environment internals, you may need other analogous approaches. For example, you could initialize the agents at the original starting line (i.e. not partway down the map) and then see which agent makes it $n$ units down the map first to determine the winner. Once a winning policy is learned by the RL agent, then gradually increase $n$ until $n$ matches the distance from the starting line to the finish line. Since it seems like you have `distance traveled` and `distance to the opponent` as features, you may instead try this method if you are unable to initialize the agents wherever you want on the map and instead can only initialize them on the starting line.
  
  A notable benefit of the overall approach is that you can more quickly debug your algorithm on the easier environments (i.e. ones with shorter episode lengths) to be confident that the learning process is correct and focus your efforts elsewhere (e.g. the training process, including the reward signal).
  
  
  ---
  
  