<p>Your work all looks correct to me. But I don't have the exercise 3.29 in my book, so I can't read the question for myself. It's not clear to me whether your last line or second to last line should be considered as the answer.</p>

CC BY-SA 4.0
2022-07-07T17:32:29.657Z
2022-07-07T17:32:29.657Z
Taw
<p>I have done exercise 3.29 from Sutton and Barto and I'd like to check if it's correct. Here's the exercise:</p>
<blockquote>
<p>Rewrite the Bellman equation for the function <span class="math-container">$q_{\pi}$</span> in terms of the three argument function <span class="math-container">$p$</span>  (3.4) and the two-argument function <span class="math-container">$r$</span>  (3.5).</p>
</blockquote>
<p>Here are the 2 functions:</p>
<p>(3.4) :</p>
<p><span class="math-container">$$p(s'|s,a) = P(S_t=s'|S_{t-1}=s,A_{t-1}=a)=\sum_{r\in\mathcal{R}}p(s',r|s,a)$$</span></p>
<p>3.5:</p>
<p><span class="math-container">$$r(s,a) = E[R_t|S_{t-1}=s,A_{t-1}=a]=\sum_{r\in\mathcal{R}}r\sum_{s'\in\mathcal{S}}p(s',r|s,a)$$</span></p>
<p>Here's my solution. Is this derivation correct?</p>
<p><span class="math-container">$$q_\pi(s,a)  = \\ E_{\pi}[G_{t+1}|S_t=s,A_t=a]= E_{\pi}[R_{t+1} + \gamma G_{t+1} |S_t=s,A_t=a]= \\ E_[R_{t+1}|S_t=s,A_t=a]+\gamma E_{\pi}[G_{t+1}|S_t=s,A_t=a] = \\ \sum_{s',r}p(s',r|s,a)r+\gamma \sum_{s'}p(s'|s,a)v_{\pi}(s') = \\ r(s,a) + \gamma \sum_{s'} p(s'|s,a) \sum_{a'}q(a'|s')\pi(a'|s')$$</span></p>

<reinforcement-learning><value-functions><sutton-barto><bellman-equations>
Is my derivation of the Bellman equation for $q_{\pi}$ in terms of $p(s'|s,a)$ and $r(s,a)$ correct?
user
# Is my derivation of the Bellman equation for $q_{\pi}$ in terms of $p(s'|s,a)$ and $r(s,a)$ correct?

(Asked by: username_0 on 2022-07-06)

I have done exercise 3.29 from Sutton and Barto and I'd like to check if it's correct. Here's the exercise:

> Rewrite the Bellman equation for the function $q\_{\pi}$ in terms of the three argument function $p$ (3.4) and the two-argument function $r$ (3.5).

Here are the 2 functions:

(3.4) :

$$p(s'|s,a) = P(S\_t=s'|S\_{t-1}=s,A\_{t-1}=a)=\sum\_{r\in\mathcal{R}}p(s',r|s,a)$$

3.5:

$$r(s,a) = E[R\_t|S\_{t-1}=s,A\_{t-1}=a]=\sum\_{r\in\mathcal{R}}r\sum\_{s'\in\mathcal{S}}p(s',r|s,a)$$

Here's my solution. Is this derivation correct?

$$q\_\pi(s,a) = \\ E\_{\pi}[G\_{t+1}|S\_t=s,A\_t=a]= E\_{\pi}[R\_{t+1} + \gamma G\_{t+1} |S\_t=s,A\_t=a]= \\ E\_[R\_{t+1}|S\_t=s,A\_t=a]+\gamma E\_{\pi}[G\_{t+1}|S\_t=s,A\_t=a] = \\ \sum\_{s',r}p(s',r|s,a)r+\gamma \sum\_{s'}p(s'|s,a)v\_{\pi}(s') = \\ r(s,a) + \gamma \sum\_{s'} p(s'|s,a) \sum\_{a'}q(a'|s')\pi(a'|s')$$

**Tags:** reinforcement-learning, value-functions, sutton-barto, bellman-equations


---

## 1 answers

### Answer by username_1 on 2022-07-07. Score: 1 | Accepted answer

Your work all looks correct to me. But I don't have the exercise 3.29 in my book, so I can't read the question for myself. It's not clear to me whether your last line or second to last line should be considered as the answer.


---

