Answers.Body: <blockquote>
  <p>Question: Can I write it without the subscript? So <span class="math-container">$$E_{\pi}[R_{t+1}|S_t=s,A_t=a] = E[R_{t+1}|S_t=s,A_t=a]$$</span></p>
  </blockquote>
  <p>Yes, your reasoning is sound, there is no need to condition the expectation on the policy, because the policy has no influence on the expected value of <span class="math-container">$R_{t+1}$</span> given that there is already a condition on <span class="math-container">$A_t=a$</span>, and <span class="math-container">$a$</span> is provided as an argument.</p>
  <p>This only works because you have split off the expected reward to evaluate separately. Overall, the value of <span class="math-container">$q_\pi$</span> does depend on the policy, and this is shown by using <span class="math-container">$v_\pi$</span> in the second term.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-27T19:28:25.327Z
Answers.LastActivityDate: 2022-06-27T19:34:07.047Z
Answers.LastEditDate: 2022-06-27T19:34:07.047Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>@Neil Slater In the expected answer it is said that</p>
  <p>&quot;This only works because you have split off the expected reward to evaluate separately. Overall, the value of <span class="math-container">$q_{\pi}$</span>
  does depend on the policy, and this is shown by using <span class="math-container">$v_{\pi}$</span>
  in the second term.&quot;</p>
  <p>However, the following solution (from solution manual) does not support this:</p>
  <p><a href="https://i.sstatic.net/RCr3u.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/RCr3u.png" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-16T17:49:04.210Z
Answers.LastActivityDate: 2024-02-16T17:49:04.210Z
Answers.UserDisplayName: DSPinfinity
Body: <p>I would like to solve the first question of Exercise 3.19 from Sutton and Barto:</p>
  <blockquote>
  <p>Exercise 3.19 The value of an action, <span class="math-container">$q_{\pi}(s, a)$</span>, depends on the expected next reward and
  the expected sum of the remaining rewards. Again we can think of this in terms of a
  small backup diagram, this one rooted at an action (state–action pair) and branching to
  the possible next states:</p>
  </blockquote>
  <p><a href="https://i.sstatic.net/PJi4B.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/PJi4B.png" alt="enter image description here" /></a></p>
  <blockquote>
  <p>Give the equation corresponding to this intuition and diagram for the action value,
  <span class="math-container">$q_{\pi}(s, a)$</span>, in terms of the expected next reward, <span class="math-container">$R_{t+1}$</span>, and the expected next state value,
  <span class="math-container">$v_{\pi}(S_{t+1})$</span>, given that <span class="math-container">$S_t =s$</span> and <span class="math-container">$A_t =a$</span>. This equation should include an expectation but
  not one conditioned on following the policy.</p>
  </blockquote>
  <p>I'm not sure how to write this equation so that it is not conditioned on <span class="math-container">$\pi$</span>.  It's clear to me that
  <span class="math-container">\begin{align*}
  q_{\pi}(s, a) &amp;= E_{\pi}[G_{t}|S_t=s,A_t=a]\\
  &amp; = E_{\pi}[R_{t+1} + \gamma G_{t+1}|S_t=s,A_t=a]\\
  &amp; = E_{\pi}[R_{t+1}|S_t=s,A_t=a] + E_{\pi}[\gamma G_{t+1}|S_t=s,A_t=a].
  \end{align*}</span></p>
  <p>The second term in the last expression  evaluates to (from Exercise 3.13) <span class="math-container">$$\sum_{s'}\gamma v_{\pi}(s')P(S_{t+1}=s'|A_t=a,S_t=s).$$</span></p>
  <p>Therefore, I'm left with the first term which is <span class="math-container">$E_{\pi}[R_{t+1}|S_t=s,A_t=a]$</span>.</p>
  <p><strong>Question</strong>: Can I write the term without the subscript as <span class="math-container">$$E_{\pi}[R_{t+1}|S_t=s,A_t=a] = E[R_{t+1}|S_t=s,A_t=a]?$$</span></p>
  <p>My reasoning is that the expected reward, given we know what <span class="math-container">$A_t$</span> is, does not depend on <span class="math-container">$\pi$</span>.</p>
  
Tags: <reinforcement-learning><markov-decision-process><rewards><sutton-barto><expectation>
Title: $E_{\pi}[R_{t+1}|S_t=s,A_t=a] = E[R_{t+1}|S_t=s,A_t=a]$?
UserDisplayName: user
ThreadText: # $E_{\pi}[R_{t+1}|S_t=s,A_t=a] = E[R_{t+1}|S_t=s,A_t=a]$?
  
  (Asked by: username_0 on 2022-06-27)
  
  I would like to solve the first question of Exercise 3.19 from Sutton and Barto:
  
  > Exercise 3.19 The value of an action, $q\_{\pi}(s, a)$, depends on the expected next reward and
  > the expected sum of the remaining rewards. Again we can think of this in terms of a
  > small backup diagram, this one rooted at an action (state–action pair) and branching to
  > the possible next states:
  
  [![enter image description here](https://i.sstatic.net/PJi4B.png)](https://i.sstatic.net/PJi4B.png)
  
  > Give the equation corresponding to this intuition and diagram for the action value,
  > $q\_{\pi}(s, a)$, in terms of the expected next reward, $R\_{t+1}$, and the expected next state value,
  > $v\_{\pi}(S\_{t+1})$, given that $S\_t =s$ and $A\_t =a$. This equation should include an expectation but
  > not one conditioned on following the policy.
  
  I'm not sure how to write this equation so that it is not conditioned on $\pi$. It's clear to me that
  \begin{align\*}
  q\_{\pi}(s, a) &= E\_{\pi}[G\_{t}|S\_t=s,A\_t=a]\\
  & = E\_{\pi}[R\_{t+1} + \gamma G\_{t+1}|S\_t=s,A\_t=a]\\
  & = E\_{\pi}[R\_{t+1}|S\_t=s,A\_t=a] + E\_{\pi}[\gamma G\_{t+1}|S\_t=s,A\_t=a].
  \end{align\*}
  
  The second term in the last expression evaluates to (from Exercise 3.13) $$\sum\_{s'}\gamma v\_{\pi}(s')P(S\_{t+1}=s'|A\_t=a,S\_t=s).$$
  
  Therefore, I'm left with the first term which is $E\_{\pi}[R\_{t+1}|S\_t=s,A\_t=a]$.
  
  **Question**: Can I write the term without the subscript as $$E\_{\pi}[R\_{t+1}|S\_t=s,A\_t=a] = E[R\_{t+1}|S\_t=s,A\_t=a]?$$
  
  My reasoning is that the expected reward, given we know what $A\_t$ is, does not depend on $\pi$.
  
  **Tags:** reinforcement-learning, markov-decision-process, rewards, sutton-barto, expectation
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2022-06-27. Score: 2 | Accepted answer
  
  > Question: Can I write it without the subscript? So $$E\_{\pi}[R\_{t+1}|S\_t=s,A\_t=a] = E[R\_{t+1}|S\_t=s,A\_t=a]$$
  
  Yes, your reasoning is sound, there is no need to condition the expectation on the policy, because the policy has no influence on the expected value of $R\_{t+1}$ given that there is already a condition on $A\_t=a$, and $a$ is provided as an argument.
  
  This only works because you have split off the expected reward to evaluate separately. Overall, the value of $q\_\pi$ does depend on the policy, and this is shown by using $v\_\pi$ in the second term.
  
  
  ---
  
  ### Answer by username_2 on 2024-02-16. Score: 0
  
  @username_1 In the expected answer it is said that
  
  "This only works because you have split off the expected reward to evaluate separately. Overall, the value of $q\_{\pi}$
  does depend on the policy, and this is shown by using $v\_{\pi}$
  in the second term."
  
  However, the following solution (from solution manual) does not support this:
  
  [![enter image description here](https://i.sstatic.net/RCr3u.png)](https://i.sstatic.net/RCr3u.png)
  
  
  ---
  
  