Answers.Body: <p><span class="math-container">$\frac{p(s', r \mid s, a)}{p(s' \mid s, a)}$</span> represents the probability of observing reward <span class="math-container">$r$</span> in state <span class="math-container">$s'$</span>, given that state <span class="math-container">$s'$</span> is the next state transitioned to. The equation assumes a probability distribution of rewards <span class="math-container">$r$</span> over state <span class="math-container">$s'$</span>, meaning that a different reward might be observed whenever a state transitions from <span class="math-container">$s$</span> to <span class="math-container">$s'$</span>. In most cases, if <span class="math-container">$r(s, a, s')$</span> is a deterministic reward then <span class="math-container">$p(s', r \mid s, a) = p(s' \mid s,a )$</span>. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-14T08:02:05.333Z
Answers.Text: But probability of observing reward r in state s', given that state s' is the next state transitioned to can be given by: Σ p(s',r|s,a) over set of all actions. I am not able to understand how these two expressions are equivalent.
Answers.UserDisplayName: SAGALPREET SINGH
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-14T08:05:58.697Z
Answers.Text: Actions in the case of the equation above is fixed rite ? $A_{t-1}$ = a. The equation does not talk about sum over all set of actions, but that a single action can lead to multiple rewards observed for state $s'$
Answers.UserDisplayName: calveeen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-14T08:08:19.077Z
Answers.Text: Yes, in the question, the action is fixed but in the first line of your answer you wrote that the fraction represents probability of observing reward r in state s', given that state s' is next state transitioned to. I was referring to that.
Answers.UserDisplayName: SAGALPREET SINGH
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-14T08:10:55.113Z
Answers.Text: In fact that is what my question was. Since, action is fixed the summation in Σ p(s',r | s,a) reduces simply to p(s',r | s,a), then why divide it by something?
Answers.UserDisplayName: SAGALPREET SINGH
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-14T08:38:51.053Z
Answers.Text: I see what you mean. Maybe you could consider the example whereby $p(s'|s,a) = 0.5$ and there are 2 rewards to be observed, $r_1$ and $r_2$, where $p(s',r_1|s,a) = 0.3$ and $p(s',r_2|s,a) = 0.2$. If u calculate expectations without dividing by the denominator, you would get $E(R_t) = 0.3R_1 + 0.2R_2$, which is not correct because in actual fact, you would observe $r_1$ about 0.6 of the time and $r_2$ 0.4 of the time when you transition to state $s'$ from state $s$, action $a$.
Answers.UserDisplayName: calveeen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-14T06:51:51.977Z
Answers.LastActivityDate: 2020-04-14T12:32:40.863Z
Answers.LastEditDate: 2020-04-14T12:32:40.863Z
Answers.UserDisplayName: calveeen
Answers.Body: <p>Expectation of reward after taking action <span class="math-container">$a$</span> in state <span class="math-container">$s$</span> and ending up in state <span class="math-container">$s'$</span> would simply be</p>
  <p><span class="math-container">\begin{equation}
  r(s, a, s') = \sum_{r \in R} r \cdot p(r|s, a, s')
  \end{equation}</span></p>
  <p>The problem with this is that they do not define probability distribution for rewards separately, they use joint distribution <span class="math-container">$p(s', r|s, a)$</span>, which represents probability for ending up in state <span class="math-container">$s'$</span> with reward <span class="math-container">$r$</span> after taking action <span class="math-container">$a$</span> in state <span class="math-container">$s$</span>. This probability can be separated in 2 parts using product rule</p>
  <p><span class="math-container">\begin{equation}
  p(s', r|s, a) = p(s'|s, a)\cdot p(r|s', s, a)
  \end{equation}</span></p>
  <p>which represents the probability for getting to state <span class="math-container">$s'$</span> from <span class="math-container">$(s, a)$</span>, and then probability for getting reward <span class="math-container">$r$</span> after ending up in <span class="math-container">$s'$</span>.</p>
  <p>If we define reward expectation through the joint distribution, we would have</p>
  <p><span class="math-container">\begin{align}
  r(s, a, s') &amp;= \sum_{r \in R} r \cdot p(s', r|s, a)\\
  &amp;= \sum_{r \in R} r \cdot p(s'|s, a) \cdot p(r|s', s, a)
  \end{align}</span></p>
  <p>but this would not be correct, since we have this extra <span class="math-container">$p(s'|s, a)$</span>, so we divide everything by it to get expression with only <span class="math-container">$p(r|s', s, a)$</span>.</p>
  <p>So, in the end we have</p>
  <p><span class="math-container">\begin{equation}
  r(s, a, s') = \sum_{r \in R} r \frac{p(r, s'|s, a)}{p(s'|s, a)}
  \end{equation}</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-14T09:45:25.667Z
Answers.LastActivityDate: 2022-04-18T09:18:29.257Z
Answers.LastEditDate: 2022-04-18T09:18:29.257Z
Answers.UserDisplayName: Brale
Body: <p>Sutton and Barto define the state–action–next-state reward function, <span class="math-container">$r(s, a, s')$</span>, as follows (<a href="http://incompleteideas.net/book/RLbook2020.pdf#page=71" rel="nofollow noreferrer">equation 3.6, p. 49</a>)</p>
  <p><span class="math-container">$$
  r(s, a, s^{\prime}) \doteq \mathbb{E}\left[R_{t} \mid S_{t-1}=s, A_{t-1}=a, S_{t}=s^{\prime}\right]=\sum_{r \in \mathcal{R}} r \frac{p(s^{\prime}, r \mid s, a )}{\color{red}{p(s^{\prime} \mid s, a)}}
  $$</span></p>
  <p>Why is the term <span class="math-container">$p(s' \mid s, a)$</span> required in this definition? Shouldn't the correct formula be <span class="math-container">$\sum_{r \in \mathcal{R}} r p(s^{\prime}, r \mid s, a )$</span>?</p>
  
Tags: <reinforcement-learning><definitions><markov-decision-process><reward-functions><sutton-barto>
Title: Why does the definition of the reward function $r(s, a, s')$ involve the term $p(s' \mid s, a)$?
UserDisplayName: SAGALPREET SINGH
ThreadText: # Why does the definition of the reward function $r(s, a, s')$ involve the term $p(s' \mid s, a)$?
  
  (Asked by: username_0 on 2020-04-14)
  
  Sutton and Barto define the state–action–next-state reward function, $r(s, a, s')$, as follows ([equation 3.6, p. 49](http://incompleteideas.net/book/RLbook2020.pdf#page=71))
  
  $$
  r(s, a, s^{\prime}) \doteq \mathbb{E}\left[R\_{t} \mid S\_{t-1}=s, A\_{t-1}=a, S\_{t}=s^{\prime}\right]=\sum\_{r \in \mathcal{R}} r \frac{p(s^{\prime}, r \mid s, a )}{\color{red}{p(s^{\prime} \mid s, a)}}
  $$
  
  Why is the term $p(s' \mid s, a)$ required in this definition? Shouldn't the correct formula be $\sum\_{r \in \mathcal{R}} r p(s^{\prime}, r \mid s, a )$?
  
  **Tags:** reinforcement-learning, definitions, markov-decision-process, reward-functions, sutton-barto
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-04-14. Score: 8 | Accepted answer
  
  Expectation of reward after taking action $a$ in state $s$ and ending up in state $s'$ would simply be
  
  \begin{equation}
  r(s, a, s') = \sum\_{r \in R} r \cdot p(r|s, a, s')
  \end{equation}
  
  The problem with this is that they do not define probability distribution for rewards separately, they use joint distribution $p(s', r|s, a)$, which represents probability for ending up in state $s'$ with reward $r$ after taking action $a$ in state $s$. This probability can be separated in 2 parts using product rule
  
  \begin{equation}
  p(s', r|s, a) = p(s'|s, a)\cdot p(r|s', s, a)
  \end{equation}
  
  which represents the probability for getting to state $s'$ from $(s, a)$, and then probability for getting reward $r$ after ending up in $s'$.
  
  If we define reward expectation through the joint distribution, we would have
  
  \begin{align}
  r(s, a, s') &= \sum\_{r \in R} r \cdot p(s', r|s, a)\\
  &= \sum\_{r \in R} r \cdot p(s'|s, a) \cdot p(r|s', s, a)
  \end{align}
  
  but this would not be correct, since we have this extra $p(s'|s, a)$, so we divide everything by it to get expression with only $p(r|s', s, a)$.
  
  So, in the end we have
  
  \begin{equation}
  r(s, a, s') = \sum\_{r \in R} r \frac{p(r, s'|s, a)}{p(s'|s, a)}
  \end{equation}
  
  
  ---
  
  ### Answer by username_1 on 2020-04-14. Score: 3
  
  $\frac{p(s', r \mid s, a)}{p(s' \mid s, a)}$ represents the probability of observing reward $r$ in state $s'$, given that state $s'$ is the next state transitioned to. The equation assumes a probability distribution of rewards $r$ over state $s'$, meaning that a different reward might be observed whenever a state transitions from $s$ to $s'$. In most cases, if $r(s, a, s')$ is a deterministic reward then $p(s', r \mid s, a) = p(s' \mid s,a )$.
  
  > **Comments:**
  > **username_0**: But probability of observing reward r in state s', given that state s' is the next state transitioned to can be given by: Σ p(s',r|s,a) over set of all actions. I am not able to understand how these two expressions are equivalent.  
  > **username_1**: Actions in the case of the equation above is fixed rite ? $A_{t-1}$ = a. The equation does not talk about sum over all set of actions, but that a single action can lead to multiple rewards observed for state $s'$  
  > **username_0**: Yes, in the question, the action is fixed but in the first line of your answer you wrote that the fraction represents probability of observing reward r in state s', given that state s' is next state transitioned to. I was referring to that.  
  > **username_0**: In fact that is what my question was. Since, action is fixed the summation in Σ p(s',r | s,a) reduces simply to p(s',r | s,a), then why divide it by something?  
  > **username_1**: I see what you mean. Maybe you could consider the example whereby $p(s'|s,a) = 0.5$ and there are 2 rewards to be observed, $r_1$ and $r_2$, where $p(s',r_1|s,a) = 0.3$ and $p(s',r_2|s,a) = 0.2$. If u calculate expectations without dividing by the denominator, you would get $E(R_t) = 0.3R_1 + 0.2R_2$, which is not correct because in actual fact, you would observe $r_1$ about 0.6 of the time and $r_2$ 0.4 of the time when you transition to state $s'$ from state $s$, action $a$.  
  
  ---
  
  