Answers.Body: <p>If your objective is for the agent to attain some goal (say, reaching a target), then a valid reward function is to assign a reward of 1 when the goal is attained and 0 otherwise. The problem with this reward function is that it's too <em>sparse</em>, meaning the agent has little guidance on how to modify their behavior to become better at attaining said goal, especially if the goal is hard to attain through a random policy in the first place (which is probably roughly what the agent starts with).</p>
  <p>The practice of modifying the reward function to guide the learning agent is called <em>reward shaping</em>.</p>
  <p>A good start is <a href="http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.48.345" rel="noreferrer"><em>Policy invariance under reward transformations: Theory and application to reward shaping</em></a> by Ng et al. The idea is to create a <em>reward potential</em> (see Theorem 1) on top of the existing reward. This reward potential should be an approximation of the <em>true value</em> of a given state. For instance, if you have a gridworld scenario where the goal is for the agent to reach some target square, you could create a reward potential based on the Manhattan distance to this target (without accounting for obstacles), which is an <em>approximation</em> to the true value of a given position.</p>
  <p>Intuitively, creating a reward potential that is close to the true values makes the job easier for the learning agent because it reduces the disadvantage of being myopic, and the agent more quickly gets closer to a &quot;somewhat good&quot; policy from which it is easier to crawl toward the optimal policy.</p>
  <p>Moreover, reward potentials have the property that they are <em>consistent</em> with the optimal policy. That is, the optimal policy to the <em>true</em> problem will not become suboptimal under the new, modified problem (with the new reward function).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-03T22:14:40.630Z
Answers.LastActivityDate: 2020-08-03T22:14:40.630Z
Answers.UserDisplayName: user76284
Answers.Body: <h3>Designing reward functions</h3>
  <p>Designing a reward function is sometimes straightforward, if you have knowledge of the problem. For example, consider the game of chess. You know that you have three outcomes: win (good), loss (bad), or draw (neutral). So, you could reward the agent with <span class="math-container">$+1$</span> if it wins the game, <span class="math-container">$-1$</span> if it loses, and <span class="math-container">$0$</span> if it draws (or for any other situation).</p>
  <p>However, in certain cases, the specification of the reward function can be a difficult task [<a href="https://ai.stanford.edu/%7Eang/papers/icml04-apprentice.pdf" rel="noreferrer">1</a>, <a href="https://es.mathworks.com/help/reinforcement-learning/ug/define-reward-signals.html" rel="noreferrer">2</a>, <a href="http://incompleteideas.net/book/RLbook2020.pdf#page=491" rel="noreferrer">3</a>] because there are many (often unknown) factors that could affect the performance of the RL agent. For example, consider the driving task, i.e. you want to teach an agent to drive e.g. a car. In this scenario, there are so many factors that affect the behavior of a driver. How can we incorporate and combine these factors in a reward function? How do we deal with unknown factors?</p>
  <p>So, often, designing a reward function is a <strong>trial-and-error</strong> and engineering process (so there is no magic formula that tells you how to design a reward function in all cases). More precisely, you define an initial reward function based on your knowledge of the problem, you observe how the agent performs, then tweak the reward function to achieve greater performance (for example, in terms of observable behavior, so <strong>not</strong> in terms of the collected reward; otherwise, this would be an easy problem: you could just design a reward function that gives infinite reward to the agent in all situations!). For example, if you have trained an RL agent to play chess, maybe you observed that the agent took a lot of time to converge (i.e. find the best policy to play the game), so you could design a new reward function that <a href="https://ai.stackexchange.com/q/24375/2444">penalizes the agent for every non-win move (maybe it will hurry up!)</a>.</p>
  <p>Of course, this trial-and-error approach is not ideal, and it can sometimes be impractical (because maybe it takes a lot of time to train the agent) and lead to misspecified reward signals.</p>
  <h3>Misspecification of rewards</h3>
  <p>It is well known that the misspecification of the reward function can have unintended and even dangerous consequences [<a href="https://openai.com/blog/faulty-reward-functions/" rel="noreferrer">5</a>]. To overcome the misspecification of rewards or improve the reward functions, you have some options, such as</p>
  <ol>
  <li><p><strong>Learning from demonstrations</strong> (aka <em>apprenticeship learning</em>), i.e. do not specify the reward function directly, but let the RL agent imitate another agent's behavior, either to</p>
  <ul>
  <li>learn the policy directly (known as <strong>imitation learning</strong> [<a href="https://papers.nips.cc/paper/2016/file/cc7e2b878868cbae992d1fb743995d8f-Paper.pdf" rel="noreferrer">8</a>]), or</li>
  <li>learn a reward function first to later learn the policy (known as <strong>inverse reinforcement learning</strong> [<a href="https://ai.stanford.edu/%7Eang/papers/icml04-apprentice.pdf" rel="noreferrer">1</a>] or sometimes known as <strong>reward learning</strong>)</li>
  </ul>
  </li>
  <li><p>Incorporate <strong>human feedback</strong> [<a href="https://openai.com/blog/deep-reinforcement-learning-from-human-preferences/" rel="noreferrer">9</a>] in the RL algorithms (in an interactive manner)</p>
  </li>
  <li><p><strong>Transfer the information</strong> in the policy learned in another but similar environment to your environment (i.e. use some kind of transfer learning for RL [<a href="https://www.jmlr.org/papers/volume10/taylor09a/taylor09a.pdf" rel="noreferrer">10</a>])</p>
  </li>
  </ol>
  <p>Of course, these solutions or approaches can also have their shortcomings. For example, interactive human feedback can be tedious.</p>
  <h3>Reward shaping</h3>
  <p>Regarding the common pitfalls, although <strong>reward shaping</strong> (i.e. augment the natural reward function with more rewards) is often suggested as a way to improve the convergence of RL algorithms, [<a href="https://hal.archives-ouvertes.fr/file/index/docid/331752/filename/matignon2006ann.pdf" rel="noreferrer">4</a>] states that reward shaping (and progress estimators) should be used cautiously.  If you want to perform reward shaping, you should probably be using <a href="https://people.eecs.berkeley.edu/%7Epabbeel/cs287-fa09/readings/NgHaradaRussell-shaping-ICML1999.pdf" rel="noreferrer"><strong>potential-based reward shaping</strong></a> (which is guaranteed not to change the optimal policy).</p>
  <h3>Further reading</h3>
  <p>The MathWorks' article <a href="https://es.mathworks.com/help/reinforcement-learning/ug/define-reward-signals.html" rel="noreferrer">Define Reward Signals</a> discusses <strong>continuous</strong> and <strong>discrete</strong> reward functions (this is also discussed in [<a href="https://hal.archives-ouvertes.fr/file/index/docid/331752/filename/matignon2006ann.pdf" rel="noreferrer">4</a>]), and addresses some of their advantages and disadvantages.</p>
  <p>Last but not least, the 2nd edition of the RL bible contains a section (<a href="http://incompleteideas.net/book/RLbook2020.pdf#page=491" rel="noreferrer">17.4 Designing Reward Signals</a>) completely dedicated to this topic.</p>
  <p>Another similar question was also asked <a href="https://ai.stackexchange.com/q/12264/2444">here</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-03T22:13:30.340Z
Answers.LastActivityDate: 2021-01-20T12:55:33.830Z
Answers.LastEditDate: 2021-01-20T12:55:33.830Z
Answers.UserDisplayName: nbro
Body: <p>Generally speaking, is there a best-practice procedure to follow when trying to define a reward function for a reinforcement-learning agent? What common pitfalls are there when defining the reward function, and how should you avoid them? What information from your problem should you take into consideration when going about it?</p>
  <p>Let us presume that our environment is fully observable MDP.</p>
  
Tags: <reinforcement-learning><reward-design><reward-functions><reward-shaping><inverse-rl>
Title: What are some best practices when trying to design a reward function?
UserDisplayName: naisuu42
ThreadText: # What are some best practices when trying to design a reward function?
  
  (Asked by: username_0 on 2020-08-03)
  
  Generally speaking, is there a best-practice procedure to follow when trying to define a reward function for a reinforcement-learning agent? What common pitfalls are there when defining the reward function, and how should you avoid them? What information from your problem should you take into consideration when going about it?
  
  Let us presume that our environment is fully observable MDP.
  
  **Tags:** reinforcement-learning, reward-design, reward-functions, reward-shaping, inverse-rl
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-08-03. Score: 9 | Accepted answer
  
  ### Designing reward functions
  
  Designing a reward function is sometimes straightforward, if you have knowledge of the problem. For example, consider the game of chess. You know that you have three outcomes: win (good), loss (bad), or draw (neutral). So, you could reward the agent with $+1$ if it wins the game, $-1$ if it loses, and $0$ if it draws (or for any other situation).
  
  However, in certain cases, the specification of the reward function can be a difficult task [[1](https://ai.stanford.edu/%7Eang/papers/icml04-apprentice.pdf), [2](https://es.mathworks.com/help/reinforcement-learning/ug/define-reward-signals.html), [3](http://incompleteideas.net/book/RLbook2020.pdf#page=491)] because there are many (often unknown) factors that could affect the performance of the RL agent. For example, consider the driving task, i.e. you want to teach an agent to drive e.g. a car. In this scenario, there are so many factors that affect the behavior of a driver. How can we incorporate and combine these factors in a reward function? How do we deal with unknown factors?
  
  So, often, designing a reward function is a **trial-and-error** and engineering process (so there is no magic formula that tells you how to design a reward function in all cases). More precisely, you define an initial reward function based on your knowledge of the problem, you observe how the agent performs, then tweak the reward function to achieve greater performance (for example, in terms of observable behavior, so **not** in terms of the collected reward; otherwise, this would be an easy problem: you could just design a reward function that gives infinite reward to the agent in all situations!). For example, if you have trained an RL agent to play chess, maybe you observed that the agent took a lot of time to converge (i.e. find the best policy to play the game), so you could design a new reward function that [penalizes the agent for every non-win move (maybe it will hurry up!)](https://ai.stackexchange.com/q/24375/2444).
  
  Of course, this trial-and-error approach is not ideal, and it can sometimes be impractical (because maybe it takes a lot of time to train the agent) and lead to misspecified reward signals.
  
  ### Misspecification of rewards
  
  It is well known that the misspecification of the reward function can have unintended and even dangerous consequences [[5](https://openai.com/blog/faulty-reward-functions/)]. To overcome the misspecification of rewards or improve the reward functions, you have some options, such as
  
  1. **Learning from demonstrations** (aka *apprenticeship learning*), i.e. do not specify the reward function directly, but let the RL agent imitate another agent's behavior, either to
  
     * learn the policy directly (known as **imitation learning** [[8](https://papers.nips.cc/paper/2016/file/cc7e2b878868cbae992d1fb743995d8f-Paper.pdf)]), or
     * learn a reward function first to later learn the policy (known as **inverse reinforcement learning** [[1](https://ai.stanford.edu/%7Eang/papers/icml04-apprentice.pdf)] or sometimes known as **reward learning**)
  2. Incorporate **human feedback** [[9](https://openai.com/blog/deep-reinforcement-learning-from-human-preferences/)] in the RL algorithms (in an interactive manner)
  3. **Transfer the information** in the policy learned in another but similar environment to your environment (i.e. use some kind of transfer learning for RL [[10](https://www.jmlr.org/papers/volume10/taylor09a/taylor09a.pdf)])
  
  Of course, these solutions or approaches can also have their shortcomings. For example, interactive human feedback can be tedious.
  
  ### Reward shaping
  
  Regarding the common pitfalls, although **reward shaping** (i.e. augment the natural reward function with more rewards) is often suggested as a way to improve the convergence of RL algorithms, [[4](https://hal.archives-ouvertes.fr/file/index/docid/331752/filename/matignon2006ann.pdf)] states that reward shaping (and progress estimators) should be used cautiously. If you want to perform reward shaping, you should probably be using [**potential-based reward shaping**](https://people.eecs.berkeley.edu/%7Epabbeel/cs287-fa09/readings/NgHaradaRussell-shaping-ICML1999.pdf) (which is guaranteed not to change the optimal policy).
  
  ### Further reading
  
  The MathWorks' article [Define Reward Signals](https://es.mathworks.com/help/reinforcement-learning/ug/define-reward-signals.html) discusses **continuous** and **discrete** reward functions (this is also discussed in [[4](https://hal.archives-ouvertes.fr/file/index/docid/331752/filename/matignon2006ann.pdf)]), and addresses some of their advantages and disadvantages.
  
  Last but not least, the 2nd edition of the RL bible contains a section ([17.4 Designing Reward Signals](http://incompleteideas.net/book/RLbook2020.pdf#page=491)) completely dedicated to this topic.
  
  Another similar question was also asked [here](https://ai.stackexchange.com/q/12264/2444).
  
  
  ---
  
  ### Answer by username_1 on 2020-08-03. Score: 6
  
  If your objective is for the agent to attain some goal (say, reaching a target), then a valid reward function is to assign a reward of 1 when the goal is attained and 0 otherwise. The problem with this reward function is that it's too *sparse*, meaning the agent has little guidance on how to modify their behavior to become better at attaining said goal, especially if the goal is hard to attain through a random policy in the first place (which is probably roughly what the agent starts with).
  
  The practice of modifying the reward function to guide the learning agent is called *reward shaping*.
  
  A good start is [*Policy invariance under reward transformations: Theory and application to reward shaping*](http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.48.345) by Ng et al. The idea is to create a *reward potential* (see Theorem 1) on top of the existing reward. This reward potential should be an approximation of the *true value* of a given state. For instance, if you have a gridworld scenario where the goal is for the agent to reach some target square, you could create a reward potential based on the Manhattan distance to this target (without accounting for obstacles), which is an *approximation* to the true value of a given position.
  
  Intuitively, creating a reward potential that is close to the true values makes the job easier for the learning agent because it reduces the disadvantage of being myopic, and the agent more quickly gets closer to a "somewhat good" policy from which it is easier to crawl toward the optimal policy.
  
  Moreover, reward potentials have the property that they are *consistent* with the optimal policy. That is, the optimal policy to the *true* problem will not become suboptimal under the new, modified problem (with the new reward function).
  
  
  ---
  
  