Answers.Body: <p>The reward function is up to you when you set the goals for the agent.</p>
  <ul>
  <li><p>If the goal is to score as highly as possible, before ending the game, then use the score. You may want to scale the score down if you are using neural networks, to prevent needing to handle very large error values in early phases of learning.</p>
  </li>
  <li><p>If the goal is to win the game, and you do not care about the score, then use the win/loss end result. I am not familiar with the game, but if it is possible to win the game - e.g. reach an exit - whilst not collecting all the gold, then the agent may choose to do that if it reduces the chance of losing.</p>
  </li>
  </ul>
  <p>The second option is harder for the agent to assess. You may want the current score to be one of the state variables, as the score is likely to be correlated with win/loss.</p>
  <p>Most computer games are designed around giving a numerical score as feedback for human play, with high score tables, players considered &quot;better&quot; if they get more points etc. If you want your agent to compete in the same way, then using the score directly will help achieve that goal.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-20T08:25:46.830Z
Answers.LastActivityDate: 2020-10-20T08:25:46.830Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>The reward function belongs to the environment and it is the only way the agent can explore the world given a state.</p>
  <p>If we want agents to do something specific, we must provide rewards to them in such a way that they will achieve our goals. It is thus very important that the reward function accurately indicates the exact behavior.</p>
  <p>Depending on your goal you can construct the function such that the agent will try to finish the game as fast as possible or collect the maximum score.</p>
  <p>For example, certain reward functions can cause an agent to commit suicide in order to avoid more severe punishment in form of negative reward in the future (e.g. if the step reward is very small). Or it will go the safest way without collecting gold if falling in pits punishment is very big. In other words, you should experiment with your reward function to find a tradeoff.</p>
  <p>Check out this <a href="https://www.youtube.com/watch?v=pVIFc72VYH8" rel="nofollow noreferrer">video</a> for more intuition behind it.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-20T11:42:39.557Z
Answers.LastActivityDate: 2022-09-06T13:14:08.187Z
Answers.LastEditDate: 2022-09-06T13:14:08.187Z
Answers.UserDisplayName: Aray Karjauv
Body: <p>I'm writing a DQN agent for the <a href="https://en.wikipedia.org/wiki/Hunt_the_Wumpus" rel="nofollow noreferrer">Wumpus game</a>.</p>
  <p>Is the reward function to train the Q-networks (target network and policy) the same as the score of the game, i.e. +1000 for picking up gold, -1000 for falling in pits and dying from the wumpus, -1 each move?</p>
  <p>This is naturally cumulative, in that the score changes after each action taken by the agent. Alternatively, is it just a +1 for win, -1 for a loss and 0 in all other situations?</p>
  
Tags: <reinforcement-learning><dqn><deep-rl><reward-design><reward-functions>
Title: How should I define the reward function to solve the Wumpus game with deep Q-learning?
UserDisplayName: Edwin Carlsson
ThreadText: # How should I define the reward function to solve the Wumpus game with deep Q-learning?
  
  (Asked by: username_0 on 2020-10-20)
  
  I'm writing a DQN agent for the [Wumpus game](https://en.wikipedia.org/wiki/Hunt_the_Wumpus).
  
  Is the reward function to train the Q-networks (target network and policy) the same as the score of the game, i.e. +1000 for picking up gold, -1000 for falling in pits and dying from the wumpus, -1 each move?
  
  This is naturally cumulative, in that the score changes after each action taken by the agent. Alternatively, is it just a +1 for win, -1 for a loss and 0 in all other situations?
  
  **Tags:** reinforcement-learning, dqn, deep-rl, reward-design, reward-functions
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-10-20. Score: 1
  
  The reward function is up to you when you set the goals for the agent.
  
  * If the goal is to score as highly as possible, before ending the game, then use the score. You may want to scale the score down if you are using neural networks, to prevent needing to handle very large error values in early phases of learning.
  * If the goal is to win the game, and you do not care about the score, then use the win/loss end result. I am not familiar with the game, but if it is possible to win the game - e.g. reach an exit - whilst not collecting all the gold, then the agent may choose to do that if it reduces the chance of losing.
  
  The second option is harder for the agent to assess. You may want the current score to be one of the state variables, as the score is likely to be correlated with win/loss.
  
  Most computer games are designed around giving a numerical score as feedback for human play, with high score tables, players considered "better" if they get more points etc. If you want your agent to compete in the same way, then using the score directly will help achieve that goal.
  
  
  ---
  
  ### Answer by username_2 on 2020-10-20. Score: 1
  
  The reward function belongs to the environment and it is the only way the agent can explore the world given a state.
  
  If we want agents to do something specific, we must provide rewards to them in such a way that they will achieve our goals. It is thus very important that the reward function accurately indicates the exact behavior.
  
  Depending on your goal you can construct the function such that the agent will try to finish the game as fast as possible or collect the maximum score.
  
  For example, certain reward functions can cause an agent to commit suicide in order to avoid more severe punishment in form of negative reward in the future (e.g. if the step reward is very small). Or it will go the safest way without collecting gold if falling in pits punishment is very big. In other words, you should experiment with your reward function to find a tradeoff.
  
  Check out this [video](https://www.youtube.com/watch?v=pVIFc72VYH8) for more intuition behind it.
  
  
  ---
  
  