Answers.Body: <p>Markov decision problems are usually defined with a reward function <span class="math-container">$r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$</span>, and in these cases the rewards are expected to be scalar real values. This makes reinforcement learning (RL) easier, for example when defining a policy <span class="math-container">$\pi(s,a)=\arg\max_a Q(s,a)$</span>, it is clear what is the maximum of the Q-factors in state <span class="math-container">$s$</span>.</p>
  <p>As you might have also realized, in practice however, problems often have multiple objectives that we wish to optimize at the same time. This is called multiobjective optimization and the related RL field is multiobjective reinforcement learning (MORL). If you have access to the paper <em>Liu, Xu, Hu: Multiobjective Reinforcement Learning: A Comprehensive Overview (2015)</em> you might be interested in reading it. (Edit: as Peter noted in his answer, the original version of this paper was found to be a plagiarism of various other works. Please refer to his answer for better resources.)</p>
  <p>The above-mentioned paper categorizes methods for dealing with multiple rewards into two categories:</p>
  <ul>
  <li><strong>single objective strategy</strong>, where multiple rewards are somehow aggregated into one scalar value. This can be done by giving weights to rewards, making some of the objectives a constraint and optimize the others, ranking the objectives and optimize them in order etc. (Note: in my experience, weighted sum of rewards is not a good objective as it might combine two completely unrelated objectives in a very forced way.)</li>
  <li><strong>Pareto strategy</strong>, where the goal is to find Pareto-optimal strategies or a Pareto front. In this case we keep the rewards a vector and may compute a composite Q-factor, eg.: <span class="math-container">$\bar{Q}(s,a)=[Q_1(s,a), \ldots, Q_N(s,a)]$</span> and may have to modify the <span class="math-container">$\arg\max_a$</span> function to select the maximum in a Pareto sense.</li>
  </ul>
  <p>Finally, I believe it is important to remind you that all these methods really depend on the use-case and on what you really want to achieve and that there is no one solution that fits all. Even after finding an appropriate method you might find yourself spending time tweaking hyper-parameters just so that your RL agent would do what you would like it to do in one specific scenario and do something else in a slightly different scenario. (Eg. taking over on a highway vs. taking over on a country road).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-15T08:54:42.830Z
Answers.LastActivityDate: 2021-01-15T15:45:52.660Z
Answers.LastEditDate: 2021-01-15T15:45:52.660Z
Answers.UserDisplayName: Hai Nguyen
Answers.Body: <p>Rather than the survey by Liu et al. recommended above, I'd suggest you read the following survey paper for an overview of MORL (disclaimer - I was a co-author on this, but I genuinely think it is a much more useful introduction to this area)</p>
  <blockquote>
  <p><a href="https://www.jair.org/index.php/jair/article/view/10836" rel="nofollow noreferrer">Roijers, D. M., Vamplew, P., Whiteson, S., &amp; Dazeley, R. (2013). A survey of multi-objective sequential decision-making. Journal of Artificial Intelligence Research, 48, 67-113.</a></p>
  </blockquote>
  <p>Liu et al's survey, in my opinion, doesn't do much more than list and briefly describe the MORL algorithms which existed at that point. There's no deeper analysis of the field. The original version of their paper was also retracted due to blatant plagiarism of several other authors, including myself as can be confirmed <a href="https://ieeexplore.ieee.org/document/6509978" rel="nofollow noreferrer">here</a>.</p>
  <p>Our survey provides arguments for the need for multiobjective methods by describing 3 scenarios where agents using single-objective RL may be unable to provide a satisfactory solution that matches the needs of the user. Briefly, these are</p>
  <ol>
  <li><p>the <strong>unknown weights</strong> scenario where the required trade-off between the objectives isn't known in advance, and so to be effective the agent must learn multiple policies corresponding to different trade-offs and then at run-time select the one which matches the current preferences (e.g. this can arise when the objectives correspond to different costs which vary in relative price over time);</p>
  </li>
  <li><p>the <strong>decision support</strong> scenario where scalarization of a reward vector is not viable (for example, in the case of subjective preferences, which defy explicit quantification), so the agent needs to learn a set of policies, and then present these to a user who will select their preferred option, and</p>
  </li>
  <li><p>the <strong>known weights</strong> scenario where the desired trade-off between objectives is known, but its nature is such that the returns are non-additive (i.e. if the user's utility function is non-linear), and therefore standard single-objective methods based on the Bellman equation can't be directly applied.</p>
  </li>
  </ol>
  <p>We propose a <strong>taxonomy</strong> of MORL problems in terms of the <em>number of policies</em> they require (single or multi-policy), the form of utility/scalarization function supported (linear or non-linear), and whether <em>deterministic</em> or <em>stochastic</em> policies are allowed, and relate this to the nature of the set of solutions which the MO algorithm needs to output. This taxonomy is then used to categorize existing MO planning and MORL methods.</p>
  <p>One final important contribution is identifying the distinction between maximising <em>Expected Scalarised Return (ESR)</em> or <em>Scalarised Expected Return (SER)</em>. The former is appropriate in cases where we are concerned about the results within each individual episode (for example, when treating a patient - that patient will only care about their own individual experience), while SER is appropriate if we care about the average return over multiple episodes. This has turned out to be a much more important issue than I anticipated at the time of the survey, and Diederik Roijers and his colleagues have examined it more closely since then (e.g., <a href="http://roijers.info/pub/esr_paper.pdf" rel="nofollow noreferrer">Multi-objective Reinforcement Learning
  for the Expected Utility of the Return</a>)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-15T15:43:12.880Z
Answers.Text: I didn't know the Liu paper was plagiarized. I think I will modify my answer accordingly. From now on I'll refer to your work instead in similar questions.
Answers.UserDisplayName: Hai Nguyen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-13T04:01:51.413Z
Answers.LastActivityDate: 2021-09-08T14:04:48.250Z
Answers.LastEditDate: 2021-09-08T14:04:48.250Z
Answers.UserDisplayName: Peter Vamplew
Answers.Body: <p>If you have multiple types of rewards (say, R1 and R2), then it is no longer clear what would be the optimal way to act: it can happen that one way of acting would maximize R1 and another way would maximize R2.  Therefore, optimal policies, value functions, etc., would all be undefined.  Of course, you could say that you want to maximize, for example, R1+R2, or 2R1+R2, etc.  But in that case, you're back at a scalar number again.</p>
  <p>It can still be helpful for other purposes to split up the reward into multiple components as you suggest, e.g., in a setup where you need to learn to predict these rewards.  But for the purpose of determining optimal actions, you need to boil it down into a single scalar.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-09T17:50:43.570Z
Answers.LastActivityDate: 2020-08-09T17:50:43.570Z
Answers.UserDisplayName: present
Body: <p>I'm reading Reinforcement Learning by Sutton &amp; Barto, and in section 3.2 they state that the reward in a Markov decision process is always a scalar real number. At the same time, I've heard about the problem of assigning credit to an action for a reward. Wouldn't a vector reward make it easier for an agent to understand the effect of an action? Specifically, a vector in which different components represent different aspects of the reward. For example, an agent driving a car may have one reward component for driving smoothly and one for staying in the lane (and these are independent of each other).</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-07T22:00:41.197Z
Comments.Text: In our self-driving car example, you are also assuming rewards as a scalar quantity implicitly. If the reward is a vector then the value function is also a vector (I'm not saying value function can't be a vector). Rich Sutton off policy function approximation, you may find something related to your question there.
Comments.UserDisplayName: Swakshar Deb
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-08T14:14:32.370Z
Comments.Text: See [this](https://ai.stackexchange.com/q/20731/2444) and [this](https://ai.stackexchange.com/q/16610/2444) questions too.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><rewards><reward-functions><multi-objective-rl>
Title: Why is the reward in reinforcement learning always a scalar?
UserDisplayName: Sid Mani
ThreadText: # Why is the reward in reinforcement learning always a scalar?
  
  (Asked by: username_0 on 2020-08-06)
  
  I'm reading Reinforcement Learning by Sutton & Barto, and in section 3.2 they state that the reward in a Markov decision process is always a scalar real number. At the same time, I've heard about the problem of assigning credit to an action for a reward. Wouldn't a vector reward make it easier for an agent to understand the effect of an action? Specifically, a vector in which different components represent different aspects of the reward. For example, an agent driving a car may have one reward component for driving smoothly and one for staying in the lane (and these are independent of each other).
  
  **Tags:** reinforcement-learning, rewards, reward-functions, multi-objective-rl
  > **Comments:**
  > **username_1**: In our self-driving car example, you are also assuming rewards as a scalar quantity implicitly. If the reward is a vector then the value function is also a vector (I'm not saying value function can't be a vector). Rich Sutton off policy function approximation, you may find something related to your question there.  
  > **username_2**: See [this](https://ai.stackexchange.com/q/20731/2444) and [this](https://ai.stackexchange.com/q/16610/2444) questions too.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_5 on 2020-08-09. Score: 9 | Accepted answer
  
  If you have multiple types of rewards (say, R1 and R2), then it is no longer clear what would be the optimal way to act: it can happen that one way of acting would maximize R1 and another way would maximize R2. Therefore, optimal policies, value functions, etc., would all be undefined. Of course, you could say that you want to maximize, for example, R1+R2, or 2R1+R2, etc. But in that case, you're back at a scalar number again.
  
  It can still be helpful for other purposes to split up the reward into multiple components as you suggest, e.g., in a setup where you need to learn to predict these rewards. But for the purpose of determining optimal actions, you need to boil it down into a single scalar.
  
  
  ---
  
  ### Answer by username_4 on 2021-01-13. Score: 8
  
  Rather than the survey by Liu et al. recommended above, I'd suggest you read the following survey paper for an overview of MORL (disclaimer - I was a co-author on this, but I genuinely think it is a much more useful introduction to this area)
  
  > [Roijers, D. M., Vamplew, P., Whiteson, S., & Dazeley, R. (2013). A survey of multi-objective sequential decision-making. Journal of Artificial Intelligence Research, 48, 67-113.](https://www.jair.org/index.php/jair/article/view/10836)
  
  Liu et al's survey, in my opinion, doesn't do much more than list and briefly describe the MORL algorithms which existed at that point. There's no deeper analysis of the field. The original version of their paper was also retracted due to blatant plagiarism of several other authors, including myself as can be confirmed [here](https://ieeexplore.ieee.org/document/6509978).
  
  Our survey provides arguments for the need for multiobjective methods by describing 3 scenarios where agents using single-objective RL may be unable to provide a satisfactory solution that matches the needs of the user. Briefly, these are
  
  1. the **unknown weights** scenario where the required trade-off between the objectives isn't known in advance, and so to be effective the agent must learn multiple policies corresponding to different trade-offs and then at run-time select the one which matches the current preferences (e.g. this can arise when the objectives correspond to different costs which vary in relative price over time);
  2. the **decision support** scenario where scalarization of a reward vector is not viable (for example, in the case of subjective preferences, which defy explicit quantification), so the agent needs to learn a set of policies, and then present these to a user who will select their preferred option, and
  3. the **known weights** scenario where the desired trade-off between objectives is known, but its nature is such that the returns are non-additive (i.e. if the user's utility function is non-linear), and therefore standard single-objective methods based on the Bellman equation can't be directly applied.
  
  We propose a **taxonomy** of MORL problems in terms of the *number of policies* they require (single or multi-policy), the form of utility/scalarization function supported (linear or non-linear), and whether *deterministic* or *stochastic* policies are allowed, and relate this to the nature of the set of solutions which the MO algorithm needs to output. This taxonomy is then used to categorize existing MO planning and MORL methods.
  
  One final important contribution is identifying the distinction between maximising *Expected Scalarised Return (ESR)* or *Scalarised Expected Return (SER)*. The former is appropriate in cases where we are concerned about the results within each individual episode (for example, when treating a patient - that patient will only care about their own individual experience), while SER is appropriate if we care about the average return over multiple episodes. This has turned out to be a much more important issue than I anticipated at the time of the survey, and Diederik Roijers and his colleagues have examined it more closely since then (e.g., [Multi-objective Reinforcement Learning
  for the Expected Utility of the Return](http://roijers.info/pub/esr_paper.pdf))
  
  > **Comments:**
  > **username_3**: I didn't know the Liu paper was plagiarized. I think I will modify my answer accordingly. From now on I'll refer to your work instead in similar questions.  
  
  ---
  
  ### Answer by username_3 on 2020-09-15. Score: 7
  
  Markov decision problems are usually defined with a reward function $r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$, and in these cases the rewards are expected to be scalar real values. This makes reinforcement learning (RL) easier, for example when defining a policy $\pi(s,a)=\arg\max\_a Q(s,a)$, it is clear what is the maximum of the Q-factors in state $s$.
  
  As you might have also realized, in practice however, problems often have multiple objectives that we wish to optimize at the same time. This is called multiobjective optimization and the related RL field is multiobjective reinforcement learning (MORL). If you have access to the paper *Liu, Xu, Hu: Multiobjective Reinforcement Learning: A Comprehensive Overview (2015)* you might be interested in reading it. (Edit: as Peter noted in his answer, the original version of this paper was found to be a plagiarism of various other works. Please refer to his answer for better resources.)
  
  The above-mentioned paper categorizes methods for dealing with multiple rewards into two categories:
  
  * **single objective strategy**, where multiple rewards are somehow aggregated into one scalar value. This can be done by giving weights to rewards, making some of the objectives a constraint and optimize the others, ranking the objectives and optimize them in order etc. (Note: in my experience, weighted sum of rewards is not a good objective as it might combine two completely unrelated objectives in a very forced way.)
  * **Pareto strategy**, where the goal is to find Pareto-optimal strategies or a Pareto front. In this case we keep the rewards a vector and may compute a composite Q-factor, eg.: $\bar{Q}(s,a)=[Q\_1(s,a), \ldots, Q\_N(s,a)]$ and may have to modify the $\arg\max\_a$ function to select the maximum in a Pareto sense.
  
  Finally, I believe it is important to remind you that all these methods really depend on the use-case and on what you really want to achieve and that there is no one solution that fits all. Even after finding an appropriate method you might find yourself spending time tweaking hyper-parameters just so that your RL agent would do what you would like it to do in one specific scenario and do something else in a slightly different scenario. (Eg. taking over on a highway vs. taking over on a country road).
  
  
  ---
  
  