Answers.Body: <p>Your work all looks correct to me. But I don't have the exercise 3.29 in my book, so I can't read the question for myself. It's not clear to me whether your last line or second to last line should be considered as the answer.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-07T17:32:29.657Z
Answers.LastActivityDate: 2022-07-07T17:32:29.657Z
Answers.UserDisplayName: Taw
Body: <p>I have done exercise 3.29 from Sutton and Barto and I'd like to check if it's correct. Here's the exercise:</p>
  <blockquote>
  <p>Rewrite the Bellman equation for the function <span class="math-container">$q_{\pi}$</span> in terms of the three argument function <span class="math-container">$p$</span>  (3.4) and the two-argument function <span class="math-container">$r$</span>  (3.5).</p>
  </blockquote>
  <p>Here are the 2 functions:</p>
  <p>(3.4) :</p>
  <p><span class="math-container">$$p(s'|s,a) = P(S_t=s'|S_{t-1}=s,A_{t-1}=a)=\sum_{r\in\mathcal{R}}p(s',r|s,a)$$</span></p>
  <p>3.5:</p>
  <p><span class="math-container">$$r(s,a) = E[R_t|S_{t-1}=s,A_{t-1}=a]=\sum_{r\in\mathcal{R}}r\sum_{s'\in\mathcal{S}}p(s',r|s,a)$$</span></p>
  <p>Here's my solution. Is this derivation correct?</p>
  <p><span class="math-container">$$q_\pi(s,a)  = \\ E_{\pi}[G_{t+1}|S_t=s,A_t=a]= E_{\pi}[R_{t+1} + \gamma G_{t+1} |S_t=s,A_t=a]= \\ E_[R_{t+1}|S_t=s,A_t=a]+\gamma E_{\pi}[G_{t+1}|S_t=s,A_t=a] = \\ \sum_{s',r}p(s',r|s,a)r+\gamma \sum_{s'}p(s'|s,a)v_{\pi}(s') = \\ r(s,a) + \gamma \sum_{s'} p(s'|s,a) \sum_{a'}q(a'|s')\pi(a'|s')$$</span></p>
  
Tags: <reinforcement-learning><value-functions><sutton-barto><bellman-equations>
Title: Is my derivation of the Bellman equation for $q_{\pi}$ in terms of $p(s'|s,a)$ and $r(s,a)$ correct?
UserDisplayName: user
ThreadText: # Is my derivation of the Bellman equation for $q_{\pi}$ in terms of $p(s'|s,a)$ and $r(s,a)$ correct?
  
  (Asked by: username_0 on 2022-07-06)
  
  I have done exercise 3.29 from Sutton and Barto and I'd like to check if it's correct. Here's the exercise:
  
  > Rewrite the Bellman equation for the function $q\_{\pi}$ in terms of the three argument function $p$ (3.4) and the two-argument function $r$ (3.5).
  
  Here are the 2 functions:
  
  (3.4) :
  
  $$p(s'|s,a) = P(S\_t=s'|S\_{t-1}=s,A\_{t-1}=a)=\sum\_{r\in\mathcal{R}}p(s',r|s,a)$$
  
  3.5:
  
  $$r(s,a) = E[R\_t|S\_{t-1}=s,A\_{t-1}=a]=\sum\_{r\in\mathcal{R}}r\sum\_{s'\in\mathcal{S}}p(s',r|s,a)$$
  
  Here's my solution. Is this derivation correct?
  
  $$q\_\pi(s,a) = \\ E\_{\pi}[G\_{t+1}|S\_t=s,A\_t=a]= E\_{\pi}[R\_{t+1} + \gamma G\_{t+1} |S\_t=s,A\_t=a]= \\ E\_[R\_{t+1}|S\_t=s,A\_t=a]+\gamma E\_{\pi}[G\_{t+1}|S\_t=s,A\_t=a] = \\ \sum\_{s',r}p(s',r|s,a)r+\gamma \sum\_{s'}p(s'|s,a)v\_{\pi}(s') = \\ r(s,a) + \gamma \sum\_{s'} p(s'|s,a) \sum\_{a'}q(a'|s')\pi(a'|s')$$
  
  **Tags:** reinforcement-learning, value-functions, sutton-barto, bellman-equations
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-07-07. Score: 1 | Accepted answer
  
  Your work all looks correct to me. But I don't have the exercise 3.29 in my book, so I can't read the question for myself. It's not clear to me whether your last line or second to last line should be considered as the answer.
  
  
  ---
  
  