Answers.Body: <p>What if a scalar reward is insufficient, or its unclear on how to collapse a multi-dimensional reward to a single dimension. Example, for someone eating a burger, both taste and cost are important. Agents may prioritize taste and cost differently, so its not clear on how to aggregate the two. It is also not clear on how a subjective categorical taste value can be combined with a numerical cost.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-19T00:44:42.163Z
Answers.Text: I like your perspective! However, I think the issue wouldn't be the different prioritization across agents (the reward hypothesis would fit them individually), but the impossibility of such prioritization. Example: COVID lockdowns. They keep people from getting infected but hurt the economy. We want to optimize for economy and health, but how do we add them up?
Answers.UserDisplayName: Bananin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-19T06:21:01.447Z
Answers.Text: it being /unclear/ as to how it will collapse into a single dimension value itself does not nessicate that there isn't one. or, perhaps, one can imagine there being a very complex function that considers pretty much everything imaginable into account
Answers.UserDisplayName: k.c. sayz 'k.c sayz'
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-13T04:06:31.447Z
Answers.Text: What you are describing is addressed in a field of research known as **multi-objective reinforcement learning (MORL)**, which uses vector rewards. I'd suggest you read the following survey paper for an overview of MORL (disclaimer - I was a co-author on this, but I genuinely think it is a useful introduction to this area) Roijers, D. M., Vamplew, P., Whiteson, S., & Dazeley, R. (2013). A survey of multi-objective sequential decision-making. Journal of Artificial Intelligence Research, 48, 67-113. https://www.jair.org/index.php/jair/article/view
Answers.UserDisplayName: Peter Vamplew
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-26T12:17:43.013Z
Answers.Text: I don't think that this is a counterexample to the reward hypothesis. It doesn't matter whether or not we know the optimal way of combining the objectives to achieve the optimal multi-objective or whether the combination of multiple objectives to form the scalar-based one leads to our desired behaviour. The RH just states that we can maximize a scalar-based objective to achieve some goal, but it says nothing about how to design that objective.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-18T17:23:48.657Z
Answers.LastActivityDate: 2020-08-18T17:23:48.657Z
Answers.UserDisplayName: Kumar
Answers.Body: <p>I believe that there is no clear answer to your question. It essentially boils down to whether <a href="https://en.wikipedia.org/wiki/Mathematical_universe_hypothesis" rel="nofollow noreferrer">you are a reductionist</a> – whether you believe that quantitative measurements can truly give justice to the complexity of the real world, and that a framework such as expectation maximization can losslessly capture what we care about as humans in the performing of tasks.</p>
  <p>From a non-reductionist perspective, one would be aware that almost any mathematical representation of complex real-world goals will necessarily be a proxy rather than the true goal (as many goals are not mathematically formalizable, such as what we perceive as &quot;good music&quot; or &quot;meaning&quot;), and thus the reward hypothesis is at best an approximation. Based on this, a non-reductionist's reward hypothesis could be rephrased as:</p>
  <blockquote>
  <p>that all of what we mean by goals and purposes can be <s>well thought of</s> <b>approximately operationalized (albeit at a certain domain-dependent loss)</b> as the maximization of the expected value of the cumulative sum of a received scalar signal (called reward)</p>
  </blockquote>
  <p>Clearly the original (stricter) version of the reward hypothesis does apply to some cases, such as purely-quantitative domains (e.g. maximizing $ earned on the stock market, or maximizing score in a video game), but as soon as the problem involves enough &quot;complexity&quot; (e.g. humans, or wherever you think the boundary should be), a non-reductionist would say that mathematics is clearly not fit to the task to truly capture the intended goal.</p>
  <p>More info on the reward hypothesis (as presented by <a href="https://www.littmania.com/" rel="nofollow noreferrer">Michael Littman</a> himself) is <a href="https://www.coursera.org/lecture/fundamentals-of-reinforcement-learning/michael-littman-the-reward-hypothesis-q6x0e" rel="nofollow noreferrer">here</a>. I would have added it as a comment to the question but do not have enough reputation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-02T23:51:35.950Z
Answers.Text: I think there's a typo in "Clearly this doesn't apply to all cases, such as purely-quantitative domains", i.e. why wouldn't it clearly apply to "purely-quantitative"?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-03T17:35:12.720Z
Answers.Text: It's actually not a typo – what I meant is that if your goal is purely defined within a quantitative system (the score in a game, the number of dollars in your bank account), then a scalar reward might in fact not be a loss-full reduction of your goal, but your goal itself. However, if one investigates even these cases closely, usually these goals turn out to also be simplifications of more fuzzy goals ("maximize the $ in my bank account while being ethical").
Answers.UserDisplayName: mdc
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-03T17:48:27.273Z
Answers.Text: Maybe you didn't get my point or I don't fully get what you wrote (though I think I do). You're saying that the reward hypothesis (RH) doesn't apply to all cases, such as purely quantitative domains, for example, maximizing the score of a game, but that is one of the examples where the RH usually applies (assuming that the goal is not fuzzy).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-06T21:51:31.477Z
Answers.Text: Thanks for the comment – edited my original post for clarity
Answers.UserDisplayName: mdc
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-02T06:16:41.247Z
Answers.LastActivityDate: 2020-12-06T21:51:08.987Z
Answers.LastEditDate: 2020-12-06T21:51:08.987Z
Answers.UserDisplayName: mdc
Answers.Body: <p>The closest counterexamples I can think of are cases where reward shaping is required to learn a good policy but ends up having unintended consequences.</p>
  <p>Reward shaping is usually used in cases we want to encourage a particular behavior or when the reward is sparse or when capturing exactly what you want is not straightforward or infeasible. But it is not a good practice to rely too much on it as it can have unintended consequences. A simple example of this is described here <a href="https://openai.com/blog/faulty-reward-functions/" rel="nofollow noreferrer">https://openai.com/blog/faulty-reward-functions/</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-01T16:37:24.843Z
Answers.Text: I don't understand how reward shaping contradicts the reward hypothesis. Can you clarify that?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-02T18:06:08.237Z
Answers.Text: I will try directly to answer the question you asked here. As you said in the comment any task can be modeled with rewards with some assumptions. But whether the modeling leads to the intended original task getting solved is not guaranteed. I consider this as a failure of reward modeling and hence these tasks might not fall under the reward maximization assumption.
Answers.UserDisplayName: saiRegrefree
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-02T18:09:07.360Z
Answers.Text: When I think about it, my point is actually targeting the cases where the original reward modeling doesn't work well with the existing training techniques (because of sparse reward or reward shaping). You can argue both ways I think.
Answers.UserDisplayName: saiRegrefree
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-02T18:18:38.497Z
Answers.Text: The reward hypothesis says that a goal can be thought of as the maximization of the reward. Either if you find a maximum or not of the objective function doesn't imply that you cannot model a problem as the maximization of the reward. So, I really don't think that reward shaping goes against the reward hypothesis.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-01T15:30:22.670Z
Answers.LastActivityDate: 2020-07-01T15:30:22.670Z
Answers.UserDisplayName: saiRegrefree
Answers.Body: <p>The book sets this hypothesis up by laying out a few assumptions:</p>
  <blockquote>
  <p>In reinforcement learning, the purpose or goal of the agent is formulated in terms of a special signal called the reward, passing from the environment to the agent. At each time step, the reward is a simple number.</p>
  </blockquote>
  <p>We could think about what counterexamples to those assumptions might be:</p>
  <ol>
  <li>The reward signal originates internally, instead of originating from the environment. (e.g. meditation, or abstract introspection)</li>
  <li>The signal is not received every time step, or isn't necessarily expected to be received at all. (e.g. seeking of transcendent experiences)</li>
  </ol>
  <p>What might be common for these counterexamples is that the reinforcement learning mechanism itself undergoes spontaneous change. A signal that would have been positive before the spontaneous change might now be negative. The reward landscape itself might be completely different. From the agent's perspective, it might be impossible to evaluate what changed. The agent might have a 'subconscious' secondary algorithm that introduces changes in the learning algorithm itself, in a way that's decoupled from any reward-defined behavior.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-01T16:40:03.870Z
Answers.Text: Why would point 1 contradict the reward hypothesis? It should not matter where the reward originates from. Your second point seems to be about sparse rewards, but that still doesn't contradict the reward hypothesis (I think).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-04T18:54:55.053Z
Answers.Text: It seems like @Bananin was looking for some examples that are hard to logically navigate through in the way traditional reward accumulation is. Both points come down to an inability to define or accumulate rewards. If rewards are spontaneous, or if the agent changes its accumulation mechanism between rewards, it makes things difficult to extrapolate and model.
Answers.UserDisplayName: bey
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-27T16:58:20.107Z
Answers.LastActivityDate: 2020-07-01T16:34:08.057Z
Answers.LastEditDate: 2020-07-01T16:34:08.057Z
Answers.UserDisplayName: bey
Body: <p>On Sutton and Barto's RL book, the reward hypothesis is stated as </p>
  
  <blockquote>
    <p>that all of what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative sum of a received scalar signal (called reward)</p>
  </blockquote>
  
  <p>Are there examples of tasks where the goals and purposes cannot be well thought of as the maximization of the expected value of the cumulative sum of a received scalar signal?</p>
  
  <p>All I can think of are tasks with subjective rewards, like "writing good music", but I am not convinced because maybe this is actually definable (perhaps by some super-intelligent alien) and we just aren't smart enough yet. Thus, I'm especially interested in counterexamples that <strong>logically</strong> or <strong>provably</strong> fail the hypothesis.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-19T10:18:37.447Z
Comments.Text: Well, "all of what we mean by goals and purposes" has some wriggle room which is open to interpretation and stretching the meanings of things. Subjective rewards are easy problem to fix, you just need to go to the source of subjective measurement - for example just ask a large enough cross-selection of people whether the output was "good music" and take an average. Or even just count downloads of the latest automated hit single. That is hard to do in practice, and slow perhaps, but the hypothesis says nothing about how convenient the reward signal needs to be.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-19T15:14:19.180Z
Comments.Text: @Bananin Are you looking for specific examples in the context of reinforcement learning or any example we can think of?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-19T16:58:04.493Z
Comments.Text: @Bananin you mention subjective rewards are counter examples but they are not, theyre just difficult to model. Like Neil said, you have to find the source of disambiguity: in his examples he offers a commonly used heuristic which is to find the expectation over some set of subjective representatives to define a more "objective" measure-- but regardless subjectivity can be made objective by conditioning on the subjective factor "good vs what I believe to be good" in which case you can still use the above concept.
Comments.UserDisplayName: mshlis
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-18T21:52:47.223Z
Comments.Text: By the way, the reward hypothesis was suggested by [Michael Littman](https://www.littmania.com/), according to [Barto & Sutton's book (2nd edition, p. 70)](http://incompleteideas.net/book/RLbook2020.pdf#page=92).
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><sutton-barto><reward-design><reward-functions><reward-hypothesis>
Title: Counterexamples to the reward hypothesis
UserDisplayName: Bananin
ThreadText: # Counterexamples to the reward hypothesis
  
  (Asked by: username_0 on 2019-11-19)
  
  On Sutton and Barto's RL book, the reward hypothesis is stated as
  
  > that all of what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative sum of a received scalar signal (called reward)
  
  Are there examples of tasks where the goals and purposes cannot be well thought of as the maximization of the expected value of the cumulative sum of a received scalar signal?
  
  All I can think of are tasks with subjective rewards, like "writing good music", but I am not convinced because maybe this is actually definable (perhaps by some super-intelligent alien) and we just aren't smart enough yet. Thus, I'm especially interested in counterexamples that **logically** or **provably** fail the hypothesis.
  
  **Tags:** reinforcement-learning, sutton-barto, reward-design, reward-functions, reward-hypothesis
  > **Comments:**
  > **username_1**: Well, "all of what we mean by goals and purposes" has some wriggle room which is open to interpretation and stretching the meanings of things. Subjective rewards are easy problem to fix, you just need to go to the source of subjective measurement - for example just ask a large enough cross-selection of people whether the output was "good music" and take an average. Or even just count downloads of the latest automated hit single. That is hard to do in practice, and slow perhaps, but the hypothesis says nothing about how convenient the reward signal needs to be.  
  > **username_2**: @Bananin Are you looking for specific examples in the context of reinforcement learning or any example we can think of?  
  > **username_3**: @Bananin you mention subjective rewards are counter examples but they are not, theyre just difficult to model. Like Neil said, you have to find the source of disambiguity: in his examples he offers a commonly used heuristic which is to find the expectation over some set of subjective representatives to define a more "objective" measure-- but regardless subjectivity can be made objective by conditioning on the subjective factor "good vs what I believe to be good" in which case you can still use the above concept.  
  > **username_2**: By the way, the reward hypothesis was suggested by [Michael Littman](https://www.littmania.com/), according to [Barto & Sutton's book (2nd edition, p. 70)](http://incompleteideas.net/book/RLbook2020.pdf#page=92).  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_4 on 2020-08-18. Score: 4
  
  What if a scalar reward is insufficient, or its unclear on how to collapse a multi-dimensional reward to a single dimension. Example, for someone eating a burger, both taste and cost are important. Agents may prioritize taste and cost differently, so its not clear on how to aggregate the two. It is also not clear on how a subjective categorical taste value can be combined with a numerical cost.
  
  > **Comments:**
  > **username_0**: I like your perspective! However, I think the issue wouldn't be the different prioritization across agents (the reward hypothesis would fit them individually), but the impossibility of such prioritization. Example: COVID lockdowns. They keep people from getting infected but hurt the economy. We want to optimize for economy and health, but how do we add them up?  
  > **username_5**: it being /unclear/ as to how it will collapse into a single dimension value itself does not nessicate that there isn't one. or, perhaps, one can imagine there being a very complex function that considers pretty much everything imaginable into account  
  > **username_6**: What you are describing is addressed in a field of research known as **multi-objective reinforcement learning (MORL)**, which uses vector rewards. I'd suggest you read the following survey paper for an overview of MORL (disclaimer - I was a co-author on this, but I genuinely think it is a useful introduction to this area) Roijers, D. M., Vamplew, P., Whiteson, S., & Dazeley, R. (2013). A survey of multi-objective sequential decision-making. Journal of Artificial Intelligence Research, 48, 67-113. https://www.jair.org/index.php/jair/article/view  
  > **username_2**: I don't think that this is a counterexample to the reward hypothesis. It doesn't matter whether or not we know the optimal way of combining the objectives to achieve the optimal multi-objective or whether the combination of multiple objectives to form the scalar-based one leads to our desired behaviour. The RH just states that we can maximize a scalar-based objective to achieve some goal, but it says nothing about how to design that objective.  
  
  ---
  
  ### Answer by username_7 on 2020-12-02. Score: 2
  
  I believe that there is no clear answer to your question. It essentially boils down to whether [you are a reductionist](https://en.wikipedia.org/wiki/Mathematical_universe_hypothesis) – whether you believe that quantitative measurements can truly give justice to the complexity of the real world, and that a framework such as expectation maximization can losslessly capture what we care about as humans in the performing of tasks.
  
  From a non-reductionist perspective, one would be aware that almost any mathematical representation of complex real-world goals will necessarily be a proxy rather than the true goal (as many goals are not mathematically formalizable, such as what we perceive as "good music" or "meaning"), and thus the reward hypothesis is at best an approximation. Based on this, a non-reductionist's reward hypothesis could be rephrased as:
  
  > that all of what we mean by goals and purposes can be ~~well thought of~~ **approximately operationalized (albeit at a certain domain-dependent loss)** as the maximization of the expected value of the cumulative sum of a received scalar signal (called reward)
  
  Clearly the original (stricter) version of the reward hypothesis does apply to some cases, such as purely-quantitative domains (e.g. maximizing $ earned on the stock market, or maximizing score in a video game), but as soon as the problem involves enough "complexity" (e.g. humans, or wherever you think the boundary should be), a non-reductionist would say that mathematics is clearly not fit to the task to truly capture the intended goal.
  
  More info on the reward hypothesis (as presented by [Michael Littman](https://www.littmania.com/) himself) is [here](https://www.coursera.org/lecture/fundamentals-of-reinforcement-learning/michael-littman-the-reward-hypothesis-q6x0e). I would have added it as a comment to the question but do not have enough reputation.
  
  > **Comments:**
  > **username_2**: I think there's a typo in "Clearly this doesn't apply to all cases, such as purely-quantitative domains", i.e. why wouldn't it clearly apply to "purely-quantitative"?  
  > **username_7**: It's actually not a typo – what I meant is that if your goal is purely defined within a quantitative system (the score in a game, the number of dollars in your bank account), then a scalar reward might in fact not be a loss-full reduction of your goal, but your goal itself. However, if one investigates even these cases closely, usually these goals turn out to also be simplifications of more fuzzy goals ("maximize the $ in my bank account while being ethical").  
  > **username_2**: Maybe you didn't get my point or I don't fully get what you wrote (though I think I do). You're saying that the reward hypothesis (RH) doesn't apply to all cases, such as purely quantitative domains, for example, maximizing the score of a game, but that is one of the examples where the RH usually applies (assuming that the goal is not fuzzy).  
  > **username_7**: Thanks for the comment – edited my original post for clarity  
  
  ---
  
  ### Answer by username_8 on 2020-07-01. Score: 0
  
  The closest counterexamples I can think of are cases where reward shaping is required to learn a good policy but ends up having unintended consequences.
  
  Reward shaping is usually used in cases we want to encourage a particular behavior or when the reward is sparse or when capturing exactly what you want is not straightforward or infeasible. But it is not a good practice to rely too much on it as it can have unintended consequences. A simple example of this is described here <https://openai.com/blog/faulty-reward-functions/>.
  
  > **Comments:**
  > **username_2**: I don't understand how reward shaping contradicts the reward hypothesis. Can you clarify that?  
  > **username_8**: I will try directly to answer the question you asked here. As you said in the comment any task can be modeled with rewards with some assumptions. But whether the modeling leads to the intended original task getting solved is not guaranteed. I consider this as a failure of reward modeling and hence these tasks might not fall under the reward maximization assumption.  
  > **username_8**: When I think about it, my point is actually targeting the cases where the original reward modeling doesn't work well with the existing training techniques (because of sparse reward or reward shaping). You can argue both ways I think.  
  > **username_2**: The reward hypothesis says that a goal can be thought of as the maximization of the reward. Either if you find a maximum or not of the objective function doesn't imply that you cannot model a problem as the maximization of the reward. So, I really don't think that reward shaping goes against the reward hypothesis.  
  
  ---
  
  ### Answer by username_9 on 2020-06-27. Score: -2
  
  The book sets this hypothesis up by laying out a few assumptions:
  
  > In reinforcement learning, the purpose or goal of the agent is formulated in terms of a special signal called the reward, passing from the environment to the agent. At each time step, the reward is a simple number.
  
  We could think about what counterexamples to those assumptions might be:
  
  1. The reward signal originates internally, instead of originating from the environment. (e.g. meditation, or abstract introspection)
  2. The signal is not received every time step, or isn't necessarily expected to be received at all. (e.g. seeking of transcendent experiences)
  
  What might be common for these counterexamples is that the reinforcement learning mechanism itself undergoes spontaneous change. A signal that would have been positive before the spontaneous change might now be negative. The reward landscape itself might be completely different. From the agent's perspective, it might be impossible to evaluate what changed. The agent might have a 'subconscious' secondary algorithm that introduces changes in the learning algorithm itself, in a way that's decoupled from any reward-defined behavior.
  
  > **Comments:**
  > **username_2**: Why would point 1 contradict the reward hypothesis? It should not matter where the reward originates from. Your second point seems to be about sparse rewards, but that still doesn't contradict the reward hypothesis (I think).  
  > **username_9**: It seems like @Bananin was looking for some examples that are hard to logically navigate through in the way traditional reward accumulation is. Both points come down to an inability to define or accumulate rewards. If rewards are spontaneous, or if the agent changes its accumulation mechanism between rewards, it makes things difficult to extrapolate and model.  
  
  ---
  
  