Answers.Body: <p>Your first option is correct:</p>
  
  <p><span class="math-container">$$r(s,a) = \mathbb{E}\left[R_t|S_{t-1}=s,A_{t-1}=a\right]=\sum_{r\in \mathcal{R}}\left[r\sum_{s'\in \mathcal{S}}p(s',r|s,a)\right]$$</span></p>
  
  <p>It's partly a matter of taste, but I prefer not moving the <span class="math-container">$r$</span> into the double sum, because its value does not change in the "inner loop". There is a small amount of intuition to be had that way around, especially when it comes to implementation (it is one multiplication after the sum, as opposed to many within the sum).</p>
  
  <p>There are a lot of sums containing sums in Sutton &amp; Barto, and they mainly follow the convention of not using any parentheses or brackets to show the one sum containing the other explicitly.</p>
  
  <p>In this case, the formulae help link to other treatments of RL, which work with the expected reward functions <span class="math-container">$r(s,a)$</span> or <span class="math-container">$r(s,a,s')$</span>, or reward matrices <span class="math-container">$R_s^a$</span>, <span class="math-container">$R_{ss'}^a$</span> such as the first edition of Sutton &amp; Barto's book. The second edition of the book uses <span class="math-container">$p(s', r|s, a)$</span> almost everywhere though, and you won't see <span class="math-container">$r(s,a)$</span> mentioned much again. So it's not worth getting too concerned about how it is presented and what the author might be saying with the presentation.</p>
  
  <p>Generally you don't need to know the <em>distribution</em> of reward, just its expectation (and how that depends on <span class="math-container">$s, a, s'$</span>), in order to derive and explain most of the results in RL. So using <span class="math-container">$r(s,a)$</span> and similar functions is fine, in places like the Bellman equations. However, the use of <span class="math-container">$p(s', r|s, a)$</span> is general without needing to bring in more functions describing the MDP.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-18T11:34:05.143Z
Answers.LastActivityDate: 2018-10-18T11:34:05.143Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am new to RL and I am trying to work through the book Reinforcement Learning: An Introduction I (Sutton &amp; Barto, 2018). In chapter 3 on Finite Markov Decision Processes, the authors write the expected reward as</p>
  
  <p><span class="math-container">$$r(s,a) = \mathbb{E}\left[R_t|S_{t-1}=s,A_{t-1}=a\right]=\sum_{r\in \mathcal{R}}r\sum_{s'\in \mathcal{S}}p(s',r|s,a)$$</span></p>
  
  <p>I am not sure if the authors mean</p>
  
  <p><span class="math-container">$$r(s,a) = \mathbb{E}\left[R_t|S_{t-1}=s,A_{t-1}=a\right]=\sum_{r\in \mathcal{R}}\left[r\sum_{s'\in \mathcal{S}}p(s',r|s,a)\right]$$</span></p>
  
  <p>or </p>
  
  <p><span class="math-container">$$r(s,a) = \mathbb{E}\left[R_t|S_{t-1}=s,A_{t-1}=a\right]=\left[\sum_{r\in \mathcal{R}}r\right]\cdot\left[\sum_{s'\in \mathcal{S}}p(s',r|s,a)\right].$$</span></p>
  
  <p>If the authors mean the first, is there any reason why it is not written like the following?</p>
  
  <p><span class="math-container">$$r(s,a) = \mathbb{E}\left[R_t|S_{t-1}=s,A_{t-1}=a\right]=\sum_{r\in \mathcal{R}}\sum_{s'\in \mathcal{S}}\left[r\,p(s',r|s,a)\right]$$</span></p>
  
Tags: <reinforcement-learning><definitions><notation><reward-functions><sutton-barto>
Title: Where are the parentheses in the definition of $r(s,a)$?
UserDisplayName: MrYouMath
ThreadText: # Where are the parentheses in the definition of $r(s,a)$?
  
  (Asked by: username_0 on 2018-10-18)
  
  I am new to RL and I am trying to work through the book Reinforcement Learning: An Introduction I (Sutton & Barto, 2018). In chapter 3 on Finite Markov Decision Processes, the authors write the expected reward as
  
  $$r(s,a) = \mathbb{E}\left[R\_t|S\_{t-1}=s,A\_{t-1}=a\right]=\sum\_{r\in \mathcal{R}}r\sum\_{s'\in \mathcal{S}}p(s',r|s,a)$$
  
  I am not sure if the authors mean
  
  $$r(s,a) = \mathbb{E}\left[R\_t|S\_{t-1}=s,A\_{t-1}=a\right]=\sum\_{r\in \mathcal{R}}\left[r\sum\_{s'\in \mathcal{S}}p(s',r|s,a)\right]$$
  
  or
  
  $$r(s,a) = \mathbb{E}\left[R\_t|S\_{t-1}=s,A\_{t-1}=a\right]=\left[\sum\_{r\in \mathcal{R}}r\right]\cdot\left[\sum\_{s'\in \mathcal{S}}p(s',r|s,a)\right].$$
  
  If the authors mean the first, is there any reason why it is not written like the following?
  
  $$r(s,a) = \mathbb{E}\left[R\_t|S\_{t-1}=s,A\_{t-1}=a\right]=\sum\_{r\in \mathcal{R}}\sum\_{s'\in \mathcal{S}}\left[r\,p(s',r|s,a)\right]$$
  
  **Tags:** reinforcement-learning, definitions, notation, reward-functions, sutton-barto
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-10-18. Score: 2 | Accepted answer
  
  Your first option is correct:
  
  $$r(s,a) = \mathbb{E}\left[R\_t|S\_{t-1}=s,A\_{t-1}=a\right]=\sum\_{r\in \mathcal{R}}\left[r\sum\_{s'\in \mathcal{S}}p(s',r|s,a)\right]$$
  
  It's partly a matter of taste, but I prefer not moving the $r$ into the double sum, because its value does not change in the "inner loop". There is a small amount of intuition to be had that way around, especially when it comes to implementation (it is one multiplication after the sum, as opposed to many within the sum).
  
  There are a lot of sums containing sums in Sutton & Barto, and they mainly follow the convention of not using any parentheses or brackets to show the one sum containing the other explicitly.
  
  In this case, the formulae help link to other treatments of RL, which work with the expected reward functions $r(s,a)$ or $r(s,a,s')$, or reward matrices $R\_s^a$, $R\_{ss'}^a$ such as the first edition of Sutton & Barto's book. The second edition of the book uses $p(s', r|s, a)$ almost everywhere though, and you won't see $r(s,a)$ mentioned much again. So it's not worth getting too concerned about how it is presented and what the author might be saying with the presentation.
  
  Generally you don't need to know the *distribution* of reward, just its expectation (and how that depends on $s, a, s'$), in order to derive and explain most of the results in RL. So using $r(s,a)$ and similar functions is fine, in places like the Bellman equations. However, the use of $p(s', r|s, a)$ is general without needing to bring in more functions describing the MDP.
  
  
  ---
  
  