Answers.Body: <p>No, the substitution you suggest based on Equation (3.4) is not correct because you forgot about the <span class="math-container">$\sum_{r \in \mathcal{R}}$</span> in the right-hand side Equation (3.4).</p>
  
  <p><strong>Equation (3.4)</strong> says (leaving out the middle part):</p>
  
  <p><span class="math-container">$$p(s' \vert s, a) \doteq \sum_{r \in \mathcal{R}} p(s', r \vert s, a).$$</span></p>
  
  <p>If you plug this into Equation (3.6) to substitute the denominator, you can't forget about that sum, you have to include the complete sum in the denominator. Because we have two different sums summing over all rewards in <span class="math-container">$\mathcal{R}$</span>, I'll change the symbol used in the second sum to <span class="math-container">$r'$</span> rather than <span class="math-container">$r$</span>. This yields:</p>
  
  <p><span class="math-container">$$r(s, a, s') \doteq \sum_{r \in \mathcal{R}} r \frac{p(s', r \vert s, a)}{\sum_{r' \in \mathcal{R}} p(s', r' \vert s, a)}.$$</span></p>
  
  <p>The numerator and denominator are different and will not cancel out.</p>
  
  <hr>
  
  <p>Intuitively, the numerator is just a single probability; the probability of observing a specific next state <span class="math-container">$s'$</span> and a reward <span class="math-container">$r$</span> given a specific current state <span class="math-container">$s$</span> and action <span class="math-container">$a$</span>. </p>
  
  <p>The denominator is a sum of many such probabilities, for <strong>all possible</strong> rewards <span class="math-container">$r'$</span> rather than just one specific reward <span class="math-container">$r$</span>. Because this sum "covers" all possible events for the rewards, it essentially represents simply the probability of observing state <span class="math-container">$s'$</span>  given state <span class="math-container">$s$</span> and action <span class="math-container">$a$</span>, paired with any arbitrary reward. That can more simply be denoted as <span class="math-container">$p(s' \vert s, a)$</span>.... which makes sense because that's the very thing we started with :D</p>
  
  <hr>
  
  <p>Equation (3.6) is as follows (again leaving the middle part out):</p>
  
  <p><span class="math-container">$$r(s, a, s') \doteq \sum_{r \in \mathcal{R}} r \frac{p(s', r \vert s, a)}{p(s' \vert s, a)}.$$</span></p>
  
  <p>In normal English, the left-hand side says "what reward do we expect to get (on average) if we transition from state <span class="math-container">$s$</span> to state <span class="math-container">$s'$</span> by executing action <span class="math-container">$a$</span>?"</p>
  
  <p>Such an <span class="math-container">$\color{red}{\text{expectation of a quantity }r}$</span> can always be computed by multiplying <span class="math-container">$\color{blue}{\text{the possible values that }r\text{ can take}}$</span> by <span class="math-container">$\color{orange}{\text{the probability of each particular value occurring}}$</span>, and summing up those multiplications. In math, this looks like:</p>
  
  <p><span class="math-container">$$\color{red}{r(s, a, s')} = \sum_{\color{blue}{r \in \mathcal{R}}} \color{blue}{r} \times \color{orange}{p(r \vert s, a, s')}.$$</span></p>
  
  <p>Now, there is a rule in probability that says (see Rule of Multiplication on <a href="https://stattrek.com/probability/probability-rules.aspx" rel="nofollow noreferrer">https://stattrek.com/probability/probability-rules.aspx</a>):</p>
  
  <p><span class="math-container">$$p(A, B) = p(A) \times p(B \vert A),$$</span></p>
  
  <p>so, if we take <span class="math-container">$A = s'$</span>, <span class="math-container">$B = r$</span>, and <span class="math-container">$s, a$</span> as additional givens for all probabilities, we get:</p>
  
  <p><span class="math-container">$$p(s', r \vert s, a) = p(s' \vert s, a) \times p(r \vert s, a, s').$$</span></p>
  
  <p>This can be rewritten (dividing both sides of the equation by <span class="math-container">$p(s' \vert s, a)$</span> and swapping the left-hand and right-hand sides) to:</p>
  
  <p><span class="math-container">$$p(r \vert s, a, s') = \frac{p(s', r \vert s, a)}{p(s' \vert s, a)}.$$</span></p>
  
  <p>Plugging this in for the orange term in the coloured equation above (which itself is hopefully fairly easy to understand intuitively) yields Equation (3.6).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-03T23:03:23.137Z
Answers.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/117994/discussion-on-answer-by-dennis-soemers-reward-related-formulation-in-reinforceme).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-19T15:23:10.143Z
Answers.LastActivityDate: 2018-10-19T17:36:11.363Z
Answers.LastEditDate: 2018-10-19T17:36:11.363Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>I am referring to eq. 3.6 (page 49) based on <a href="https://drive.google.com/file/d/1opPSz5AZ_kVa1uWOdOiveNiBFiEOHjkG/view" rel="nofollow noreferrer">Sutton's online book</a>  and can be found in an image below.</p>
  <p><a href="https://i.sstatic.net/VTPdz.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/VTPdz.png" alt="enter image description here" /></a></p>
  <p>I could not make sense of the final derivation of the equation <span class="math-container">$r(s, a, s')$</span>. My question is actually how do we come to that final derivation?</p>
  <p>Surprisingly, the denominator of <span class="math-container">$p(s'|s, a)$</span> can literally be replaced by <span class="math-container">$p(s', r|s, a)$</span> as eq. 3.4 suggests, then it will end up with &quot;<span class="math-container">$r$</span>&quot; term only due to cancellation of numerator <span class="math-container">$p(s', r|s, a)$</span> and denominator <span class="math-container">$p(s'|s, a)$</span>.</p>
  <p>Any explanation on that would be appreciated.</p>
  
Tags: <reinforcement-learning><math><notation><reward-functions><probability-theory>
Title: Why is the equation $r(s', a, s') =\sum_{r \in \mathcal{R}} r \frac{p\left(s^{\prime}, r \mid s, a\right)}{p\left(s^{\prime} \mid s, a\right)}$true?
UserDisplayName: alfa_80
ThreadText: # Why is the equation $r(s', a, s') =\sum_{r \in \mathcal{R}} r \frac{p\left(s^{\prime}, r \mid s, a\right)}{p\left(s^{\prime} \mid s, a\right)}$true?
  
  (Asked by: username_0 on 2018-10-19)
  
  I am referring to eq. 3.6 (page 49) based on [Sutton's online book](https://drive.google.com/file/d/1opPSz5AZ_kVa1uWOdOiveNiBFiEOHjkG/view) and can be found in an image below.
  
  [![enter image description here](https://i.sstatic.net/VTPdz.png)](https://i.sstatic.net/VTPdz.png)
  
  I could not make sense of the final derivation of the equation $r(s, a, s')$. My question is actually how do we come to that final derivation?
  
  Surprisingly, the denominator of $p(s'|s, a)$ can literally be replaced by $p(s', r|s, a)$ as eq. 3.4 suggests, then it will end up with "$r$" term only due to cancellation of numerator $p(s', r|s, a)$ and denominator $p(s'|s, a)$.
  
  Any explanation on that would be appreciated.
  
  **Tags:** reinforcement-learning, math, notation, reward-functions, probability-theory
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-10-19. Score: 5 | Accepted answer
  
  No, the substitution you suggest based on Equation (3.4) is not correct because you forgot about the $\sum\_{r \in \mathcal{R}}$ in the right-hand side Equation (3.4).
  
  **Equation (3.4)** says (leaving out the middle part):
  
  $$p(s' \vert s, a) \doteq \sum\_{r \in \mathcal{R}} p(s', r \vert s, a).$$
  
  If you plug this into Equation (3.6) to substitute the denominator, you can't forget about that sum, you have to include the complete sum in the denominator. Because we have two different sums summing over all rewards in $\mathcal{R}$, I'll change the symbol used in the second sum to $r'$ rather than $r$. This yields:
  
  $$r(s, a, s') \doteq \sum\_{r \in \mathcal{R}} r \frac{p(s', r \vert s, a)}{\sum\_{r' \in \mathcal{R}} p(s', r' \vert s, a)}.$$
  
  The numerator and denominator are different and will not cancel out.
  
  ---
  
  Intuitively, the numerator is just a single probability; the probability of observing a specific next state $s'$ and a reward $r$ given a specific current state $s$ and action $a$.
  
  The denominator is a sum of many such probabilities, for **all possible** rewards $r'$ rather than just one specific reward $r$. Because this sum "covers" all possible events for the rewards, it essentially represents simply the probability of observing state $s'$ given state $s$ and action $a$, paired with any arbitrary reward. That can more simply be denoted as $p(s' \vert s, a)$.... which makes sense because that's the very thing we started with :D
  
  ---
  
  Equation (3.6) is as follows (again leaving the middle part out):
  
  $$r(s, a, s') \doteq \sum\_{r \in \mathcal{R}} r \frac{p(s', r \vert s, a)}{p(s' \vert s, a)}.$$
  
  In normal English, the left-hand side says "what reward do we expect to get (on average) if we transition from state $s$ to state $s'$ by executing action $a$?"
  
  Such an $\color{red}{\text{expectation of a quantity }r}$ can always be computed by multiplying $\color{blue}{\text{the possible values that }r\text{ can take}}$ by $\color{orange}{\text{the probability of each particular value occurring}}$, and summing up those multiplications. In math, this looks like:
  
  $$\color{red}{r(s, a, s')} = \sum\_{\color{blue}{r \in \mathcal{R}}} \color{blue}{r} \times \color{orange}{p(r \vert s, a, s')}.$$
  
  Now, there is a rule in probability that says (see Rule of Multiplication on <https://stattrek.com/probability/probability-rules.aspx>):
  
  $$p(A, B) = p(A) \times p(B \vert A),$$
  
  so, if we take $A = s'$, $B = r$, and $s, a$ as additional givens for all probabilities, we get:
  
  $$p(s', r \vert s, a) = p(s' \vert s, a) \times p(r \vert s, a, s').$$
  
  This can be rewritten (dividing both sides of the equation by $p(s' \vert s, a)$ and swapping the left-hand and right-hand sides) to:
  
  $$p(r \vert s, a, s') = \frac{p(s', r \vert s, a)}{p(s' \vert s, a)}.$$
  
  Plugging this in for the orange term in the coloured equation above (which itself is hopefully fairly easy to understand intuitively) yields Equation (3.6).
  
  > **Comments:**
  > **username_2**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/117994/discussion-on-answer-by-username_1-username_1-reward-related-formulation-in-reinforceme).  
  
  ---
  
  