Answers.Body: <p>There are a few different, but equivalent, ways to express the relationships between value functions in the Bellman equations.</p>
  <p>Some differences are just notation, but in the case of the two equations that you give, there is a more practical difference.</p>
  <p>Specifcally, the term <span class="math-container">$R(s,a)$</span> from the second equation that you give is the <em>expected</em> reward from taking action <span class="math-container">$a$</span> in state <span class="math-container">$s$</span>. Even though individual rewards may depend on <span class="math-container">$s'$</span>, the expectation does not for a consistent MDP.</p>
  <p>You can see that the two equations are equivalent, when you consider that:</p>
  <p><span class="math-container">$$R(s,a) = \sum_{r,s'} r p(r,s'|s,a)$$</span></p>
  <p>and</p>
  <p><span class="math-container">$$P(s'|s,a) = \sum_r p(r,s'|s,a)$$</span></p>
  <p>The first equation for me is the more intuitive version of the state value Bellman equation as it explicitly shows all the dependencies for the transition. The second equation can be more practical in some circumstances, because if you know or easily calculate <span class="math-container">$R(s,a)$</span> for any state/action pair, then it is often more convenient to use it than recalculate each time from first principles. In addition, <span class="math-container">$R(s,a)$</span> is often under the control of the developer, and may well be some constant or fixed simple equation based on <span class="math-container">$s$</span> and <span class="math-container">$a$</span>, in which case the <span class="math-container">$R(s,a)$</span> notation captures that exactly.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-01T10:56:32.877Z
Answers.Text: This is correct but not that this is just the definition of the state value function, so I don't think that the practical aspects is very important here, but I understand what you mean. It may also be worth noting that, in many cases, the reward is deterministic, so $r(s, a)$ will just be the expectation of a constant, which is a constant itself, i.e. $r(s, a)$ can be replaced with a deterministic reward function rather than being an expectation.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-01T13:56:13.763Z
Answers.Text: @nbro: I would say that the Bellman equations are not the same as the state value definition. The definition IMO is $v_{\pi}(s) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^k R_{t+1+k} | S_{t} = s]$ and the Bellman equations in the question are derived from it.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-01T08:01:44.550Z
Answers.LastActivityDate: 2022-03-01T13:53:16.147Z
Answers.LastEditDate: 2022-03-01T13:53:16.147Z
Answers.UserDisplayName: Neil Slater
Body: <p>Why are there different forms of the value function in reinforcement learning?</p>
  <p>Sutton &amp; Barto (<a href="http://incompleteideas.net/book/RLbook2020.pdf#page=81" rel="nofollow noreferrer">2nd edition, equation 3.14</a>) define the state value function as follows</p>
  <p><span class="math-container">$$v_{\pi}(s) = \displaystyle\sum_a \pi(a|s) \displaystyle\sum_{s',r}p(s',r \mid s,a)[r + \gamma v_\pi(s')].$$</span></p>
  <p>Many lecture notes (for example, <a href="https://canvas.harvard.edu/files/3870466/download?download_frd=1&amp;verifier=7WRDGA5sRbeCdbtZ90VF0mlzfaCBiXc8NyBJ90q9" rel="nofollow noreferrer">these</a> by Haim Sompolinsky, MCB 131, 2017) define it as follows (equation 18)</p>
  <p><span class="math-container">$$V^{\pi}(s) =\displaystyle\sum_a \pi(a,s) [R(s,a) + \gamma \displaystyle\sum_{s'}P(s' \mid s,a) V^\pi(s')].$$</span></p>
  <p>Are these two forms the same?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-28T12:06:40.483Z
Comments.Text: Please, focus on one question. You're asking at least 2 distinct questions here. Also, provide the links to the slides from Carnegie Mellon. I think your formulas have been copied incorrectly or imprecisely. Definitely, the notation is not fully correct.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-28T13:38:27.917Z
Comments.Text: Have I seen this question before? In any case, the first equation contains some mistakes (no declaration of $r$, it should be in the second sum, and $r_{t+1}$ appears when there are no time steps - should be just $r$). Both equations seem to skip superscript on $V^{\pi}$. It would be good to fix these, and/or give full references to where the equations are from (Sutton&Barto numbers its key equations, so you could say exactly which one you meant to copy). I don't want to assume anything and "fix" equations myself when the question is directly about them.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-01T03:43:16.913Z
Comments.Text: I edited the question so that it reflects the sources of the equations.
Comments.UserDisplayName: cgo
Tags: <reinforcement-learning><markov-decision-process><proofs><value-functions>
Title: Are these two forms of the state value function the same?
UserDisplayName: cgo
ThreadText: # Are these two forms of the state value function the same?
  
  (Asked by: username_0 on 2022-02-28)
  
  Why are there different forms of the value function in reinforcement learning?
  
  Sutton & Barto ([2nd edition, equation 3.14](http://incompleteideas.net/book/RLbook2020.pdf#page=81)) define the state value function as follows
  
  $$v\_{\pi}(s) = \displaystyle\sum\_a \pi(a|s) \displaystyle\sum\_{s',r}p(s',r \mid s,a)[r + \gamma v\_\pi(s')].$$
  
  Many lecture notes (for example, [these](https://canvas.harvard.edu/files/3870466/download?download_frd=1&verifier=7WRDGA5sRbeCdbtZ90VF0mlzfaCBiXc8NyBJ90q9) by Haim Sompolinsky, MCB 131, 2017) define it as follows (equation 18)
  
  $$V^{\pi}(s) =\displaystyle\sum\_a \pi(a,s) [R(s,a) + \gamma \displaystyle\sum\_{s'}P(s' \mid s,a) V^\pi(s')].$$
  
  Are these two forms the same?
  
  **Tags:** reinforcement-learning, markov-decision-process, proofs, value-functions
  > **Comments:**
  > **username_1**: Please, focus on one question. You're asking at least 2 distinct questions here. Also, provide the links to the slides from Carnegie Mellon. I think your formulas have been copied incorrectly or imprecisely. Definitely, the notation is not fully correct.  
  > **username_2**: Have I seen this question before? In any case, the first equation contains some mistakes (no declaration of $r$, it should be in the second sum, and $r_{t+1}$ appears when there are no time steps - should be just $r$). Both equations seem to skip superscript on $V^{\pi}$. It would be good to fix these, and/or give full references to where the equations are from (Sutton&Barto numbers its key equations, so you could say exactly which one you meant to copy). I don't want to assume anything and "fix" equations myself when the question is directly about them.  
  > **username_0**: I edited the question so that it reflects the sources of the equations.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-03-01. Score: 4
  
  There are a few different, but equivalent, ways to express the relationships between value functions in the Bellman equations.
  
  Some differences are just notation, but in the case of the two equations that you give, there is a more practical difference.
  
  Specifcally, the term $R(s,a)$ from the second equation that you give is the *expected* reward from taking action $a$ in state $s$. Even though individual rewards may depend on $s'$, the expectation does not for a consistent MDP.
  
  You can see that the two equations are equivalent, when you consider that:
  
  $$R(s,a) = \sum\_{r,s'} r p(r,s'|s,a)$$
  
  and
  
  $$P(s'|s,a) = \sum\_r p(r,s'|s,a)$$
  
  The first equation for me is the more intuitive version of the state value Bellman equation as it explicitly shows all the dependencies for the transition. The second equation can be more practical in some circumstances, because if you know or easily calculate $R(s,a)$ for any state/action pair, then it is often more convenient to use it than recalculate each time from first principles. In addition, $R(s,a)$ is often under the control of the developer, and may well be some constant or fixed simple equation based on $s$ and $a$, in which case the $R(s,a)$ notation captures that exactly.
  
  > **Comments:**
  > **username_1**: This is correct but not that this is just the definition of the state value function, so I don't think that the practical aspects is very important here, but I understand what you mean. It may also be worth noting that, in many cases, the reward is deterministic, so $r(s, a)$ will just be the expectation of a constant, which is a constant itself, i.e. $r(s, a)$ can be replaced with a deterministic reward function rather than being an expectation.  
  > **username_2**: @nbro: I would say that the Bellman equations are not the same as the state value definition. The definition IMO is $v_{\pi}(s) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^k R_{t+1+k} | S_{t} = s]$ and the Bellman equations in the question are derived from it.  
  
  ---
  
  