Answers.Body: <p>The two are equivalent.</p>
  <p><span class="math-container">\begin{align}
  V_\pi(s) &amp;= \sum_{a}^{}\pi(a|s)  \sum_{s',r}^{}p(s',r |s,a)[r + \gamma V_\pi(s')]\\
  &amp;= \sum_{a}^{}\pi(a|s)  \sum_{s',r}^{}p(s'|s,a)p(r| s',a,s)[r + \gamma V_\pi(s')] \\
  &amp;= \sum_{a}^{}\pi(a|s)  \sum_{s'}^{}T(s',s,a)\sum_{r}p(r| s',a,s)[r + \gamma V_\pi(s')]\\
  &amp;= \sum_{a}^{}\pi(a|s)  \sum_{s'}^{}T(s',s,a)\left[\sum_{r}p(r| s',a,s)r + \gamma \sum_{r}p(r| s',a,s)V_\pi(s')] \right]\\
  &amp;= \sum_{a}^{}\pi(a|s)  \sum_{s'}^{}T(s',s,a)\left[R(s',a,s) + \gamma V_\pi(s')] \right]\\
  \end{align}</span></p>
  <p>We first split <span class="math-container">$p(s',r|s,a) = p(s'|s,a)p(r|s',a,s)$</span> by law of conditional probability, and we note that <span class="math-container">$p(s'|s,a) = T(s',a,s)$</span> by definition. We split the sum to parts over <span class="math-container">$s'$</span> and <span class="math-container">$r$</span>, and then split it further to separate the two terms taking the expectation over <span class="math-container">$r$</span> and <span class="math-container">$\gamma V_\pi(s')$</span>. By definition, the first expectation equals <span class="math-container">$R(s',a,s)$</span>. In the second expecation, <span class="math-container">$V_\pi$</span> is independent of <span class="math-container">$r$</span> so the probability sums to one and vanishes.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-30T15:12:19.100Z
Answers.Text: Just one note for people that get confused with the last part. If $k$ is a constant and we have $k \sum_{i=1}^N f(i)$, then this is equivalent to $k f(1) + k f(2) + \dots + k f(N) = k * (f(1) + f(2) + \dots + f(N))$. In this specific case, $p$ is a distribution. So, the sum of all $f(i)$ is equal to one, so we multiply $1$ times $k$, which in this case is $V_\pi(s')$.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-02T12:04:41.057Z
Answers.LastActivityDate: 2022-11-02T12:04:41.057Z
Answers.UserDisplayName: mikkola
Body: <p>The first version is the one I am most familiar with:</p>
  <p><span class="math-container">$$V_\pi(s) = \sum_{a}^{}\pi(a|s)  \sum_{s'}^{}T(s, a, s')[R(s, a, s') + \gamma V_\pi(s')]$$</span></p>
  <p>where <span class="math-container">$T(s, a, s')$</span> represents the <strong>probability of transitioning</strong> from state <span class="math-container">$s$</span> to state <span class="math-container">$s'$</span> given action <span class="math-container">$a$</span>.
  And where <span class="math-container">$R(s, a, s')$</span> represents the <strong>expected value of the reward</strong> from transitioning from state <span class="math-container">$s$</span> to state <span class="math-container">$s'$</span> given action a.</p>
  <p>In another textbook, I saw this version of the equation:</p>
  <p><span class="math-container">$$V_\pi(s) = \sum_{a}^{}\pi(a|s)  \sum_{s',r}^{}p(s',r |s,a)[r + \gamma V_\pi(s')]$$</span></p>
  <p>Is the second equation more general since it considers the probability of going to <span class="math-container">$s'$</span> AND getting a reward <span class="math-container">$r$</span>?</p>
  <p>Or are they somehow the same (if so, why)?</p>
  <p>Or maybe they are the same under certain circumstances (like conditional independence of <span class="math-container">$s'$</span> and <span class="math-container">$r$</span>)?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-31T10:19:11.903Z
Comments.Text: @RLControl The second edition of that book uses that second form, but the first edition doesn't (if I remember correctly).
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><markov-decision-process><value-functions><bellman-equations>
Title: What is the difference between these two versions of the Bellman equation?
UserDisplayName: Nova
ThreadText: # What is the difference between these two versions of the Bellman equation?
  
  (Asked by: username_0 on 2022-11-01)
  
  The first version is the one I am most familiar with:
  
  $$V\_\pi(s) = \sum\_{a}^{}\pi(a|s) \sum\_{s'}^{}T(s, a, s')[R(s, a, s') + \gamma V\_\pi(s')]$$
  
  where $T(s, a, s')$ represents the **probability of transitioning** from state $s$ to state $s'$ given action $a$.
  And where $R(s, a, s')$ represents the **expected value of the reward** from transitioning from state $s$ to state $s'$ given action a.
  
  In another textbook, I saw this version of the equation:
  
  $$V\_\pi(s) = \sum\_{a}^{}\pi(a|s) \sum\_{s',r}^{}p(s',r |s,a)[r + \gamma V\_\pi(s')]$$
  
  Is the second equation more general since it considers the probability of going to $s'$ AND getting a reward $r$?
  
  Or are they somehow the same (if so, why)?
  
  Or maybe they are the same under certain circumstances (like conditional independence of $s'$ and $r$)?
  
  **Tags:** reinforcement-learning, markov-decision-process, value-functions, bellman-equations
  > **Comments:**
  > **username_1**: @RLControl The second edition of that book uses that second form, but the first edition doesn't (if I remember correctly).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-11-02. Score: 4 | Accepted answer
  
  The two are equivalent.
  
  \begin{align}
  V\_\pi(s) &= \sum\_{a}^{}\pi(a|s) \sum\_{s',r}^{}p(s',r |s,a)[r + \gamma V\_\pi(s')]\\
  &= \sum\_{a}^{}\pi(a|s) \sum\_{s',r}^{}p(s'|s,a)p(r| s',a,s)[r + \gamma V\_\pi(s')] \\
  &= \sum\_{a}^{}\pi(a|s) \sum\_{s'}^{}T(s',s,a)\sum\_{r}p(r| s',a,s)[r + \gamma V\_\pi(s')]\\
  &= \sum\_{a}^{}\pi(a|s) \sum\_{s'}^{}T(s',s,a)\left[\sum\_{r}p(r| s',a,s)r + \gamma \sum\_{r}p(r| s',a,s)V\_\pi(s')] \right]\\
  &= \sum\_{a}^{}\pi(a|s) \sum\_{s'}^{}T(s',s,a)\left[R(s',a,s) + \gamma V\_\pi(s')] \right]\\
  \end{align}
  
  We first split $p(s',r|s,a) = p(s'|s,a)p(r|s',a,s)$ by law of conditional probability, and we note that $p(s'|s,a) = T(s',a,s)$ by definition. We split the sum to parts over $s'$ and $r$, and then split it further to separate the two terms taking the expectation over $r$ and $\gamma V\_\pi(s')$. By definition, the first expectation equals $R(s',a,s)$. In the second expecation, $V\_\pi$ is independent of $r$ so the probability sums to one and vanishes.
  
  > **Comments:**
  > **username_1**: Just one note for people that get confused with the last part. If $k$ is a constant and we have $k \sum_{i=1}^N f(i)$, then this is equivalent to $k f(1) + k f(2) + \dots + k f(N) = k * (f(1) + f(2) + \dots + f(N))$. In this specific case, $p$ is a distribution. So, the sum of all $f(i)$ is equal to one, so we multiply $1$ times $k$, which in this case is $V_\pi(s')$.  
  
  ---
  
  