<p>Here's your equation with an additional couple of parenthesis that emphasizes the order of the operations (note that you had a small typo in your original equation).</p>
<p><span class="math-container">$$v_{\pi}(s) =\sum_a \pi(a \mid s) \left(\sum_{s',r} p(s',r \mid s,a)[r+ \gamma v_\pi(s')] \right)$$</span></p>
<p>Now, let me answer your other questions.</p>
<blockquote>
<p>Is the second sum using the index <span class="math-container">$a$</span> from the first sum?</p>
</blockquote>
<p>Yes.</p>
<blockquote>
<p>Or is it independent, and can I move out the <span class="math-container">$[r+ \gamma v_\pi(s')]$</span> term out of the sum?</p>
</blockquote>
<p>No, and you cannot move this term out of the sum because the second sum is a sum over <span class="math-container">$r$</span> and <span class="math-container">$s'$</span> and <span class="math-container">$r+ \gamma v_\pi(s')$</span> depends on those terms.</p>
<p>Note that <span class="math-container">$v_{\pi}(s)$</span> is defined as an expectation and that <span class="math-container">$\pi(a \mid s)$</span> (the policy) and <span class="math-container">$p(s',r \mid s,a)$</span> (the <a href="https://ai.stackexchange.com/a/8820/2444">model</a>) are probability distributions.</p>

CC BY-SA 4.0
2022-01-18T09:00:46.237Z
Worth noting this is a common convention when concatenating $\sum$ terms, that they are nested with implied parens as you show. I'm sure there will be some discipline that does not do that, but all the ML I have read does this.
Neil Slater
CC BY-SA 4.0
2022-01-18T09:35:13.600Z
I'm sorry, I'm coming back and got confused, shouldn't be like this:
$$v_{\pi}(s) =\sum_a \left( \pi(a \mid s) \sum_{s',r} p(s',r \mid s,a)[r+ \gamma v_\pi(s')] \right)$$
Since the second sum uses the same $a$ index.. Putting the indexes the way @nbro does implies the sums are independent
nammerkage
CC BY-SA 4.0
2022-01-18T11:47:51.640Z
@nammerkage You can also write that **and** with the parentheses that I show in my answer. Both are ok. Parentheses here are used just to emphasize the precedence of the operations. So, you can also write $$v_{\pi}(s) =\sum_a \left( \pi(a \mid s) \left( \sum_{s',r} p(s',r \mid s,a)[r+ \gamma v_\pi(s')]\right) \right)$$
nbro
CC BY-SA 4.0
2022-01-18T08:35:45.067Z
2022-01-18T08:35:45.067Z
nbro
<p>I'm not having a lot of intuition about the equation. I have this Bellman update rule:</p>
<p><span class="math-container">$$v_{\pi}(s) =\sum_a \pi(a|s)\sum_{s',r} p(s',r|s,a)[r+ \gamma v_{k}(s')]$$</span></p>
<p>But where are the parenthesis? Is the second sum using the index <span class="math-container">$a$</span> from the first sum? Or is it independent, and can I move out the <span class="math-container">$[r+ \gamma v_{k}(s')]$</span> term out of the sum?</p>

<reinforcement-learning><notation><bellman-equations>
Where are the parentheses in the Bellman update rule?
nammerkage
# Where are the parentheses in the Bellman update rule?

(Asked by: username_0 on 2022-01-18)

I'm not having a lot of intuition about the equation. I have this Bellman update rule:

$$v\_{\pi}(s) =\sum\_a \pi(a|s)\sum\_{s',r} p(s',r|s,a)[r+ \gamma v\_{k}(s')]$$

But where are the parenthesis? Is the second sum using the index $a$ from the first sum? Or is it independent, and can I move out the $[r+ \gamma v\_{k}(s')]$ term out of the sum?

**Tags:** reinforcement-learning, notation, bellman-equations


---

## 1 answers

### Answer by username_1 on 2022-01-18. Score: 3 | Accepted answer

Here's your equation with an additional couple of parenthesis that emphasizes the order of the operations (note that you had a small typo in your original equation).

$$v\_{\pi}(s) =\sum\_a \pi(a \mid s) \left(\sum\_{s',r} p(s',r \mid s,a)[r+ \gamma v\_\pi(s')] \right)$$

Now, let me answer your other questions.

> Is the second sum using the index $a$ from the first sum?

Yes.

> Or is it independent, and can I move out the $[r+ \gamma v\_\pi(s')]$ term out of the sum?

No, and you cannot move this term out of the sum because the second sum is a sum over $r$ and $s'$ and $r+ \gamma v\_\pi(s')$ depends on those terms.

Note that $v\_{\pi}(s)$ is defined as an expectation and that $\pi(a \mid s)$ (the policy) and $p(s',r \mid s,a)$ (the [model](https://ai.stackexchange.com/a/8820/2444)) are probability distributions.

> **Comments:**
> **username_2**: Worth noting this is a common convention when concatenating $\sum$ terms, that they are nested with implied parens as you show. I'm sure there will be some discipline that does not do that, but all the ML I have read does this.  
> **username_0**: I'm sorry, I'm coming back and got confused, shouldn't be like this: $$v_{\pi}(s) =\sum_a \left( \pi(a \mid s) \sum_{s',r} p(s',r \mid s,a)[r+ \gamma v_\pi(s')] \right)$$ Since the second sum uses the same $a$ index.. Putting the indexes the way @username_1 does implies the sums are independent  
> **username_1**: @username_0 You can also write that **and** with the parentheses that I show in my answer. Both are ok. Parentheses here are used just to emphasize the precedence of the operations. So, you can also write $$v_{\pi}(s) =\sum_a \left( \pi(a \mid s) \left( \sum_{s',r} p(s',r \mid s,a)[r+ \gamma v_\pi(s')]\right) \right)$$  

---

