Answers.Body: <p>Here's your equation with an additional couple of parenthesis that emphasizes the order of the operations (note that you had a small typo in your original equation).</p>
  <p><span class="math-container">$$v_{\pi}(s) =\sum_a \pi(a \mid s) \left(\sum_{s',r} p(s',r \mid s,a)[r+ \gamma v_\pi(s')] \right)$$</span></p>
  <p>Now, let me answer your other questions.</p>
  <blockquote>
  <p>Is the second sum using the index <span class="math-container">$a$</span> from the first sum?</p>
  </blockquote>
  <p>Yes.</p>
  <blockquote>
  <p>Or is it independent, and can I move out the <span class="math-container">$[r+ \gamma v_\pi(s')]$</span> term out of the sum?</p>
  </blockquote>
  <p>No, and you cannot move this term out of the sum because the second sum is a sum over <span class="math-container">$r$</span> and <span class="math-container">$s'$</span> and <span class="math-container">$r+ \gamma v_\pi(s')$</span> depends on those terms.</p>
  <p>Note that <span class="math-container">$v_{\pi}(s)$</span> is defined as an expectation and that <span class="math-container">$\pi(a \mid s)$</span> (the policy) and <span class="math-container">$p(s',r \mid s,a)$</span> (the <a href="https://ai.stackexchange.com/a/8820/2444">model</a>) are probability distributions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-18T09:00:46.237Z
Answers.Text: Worth noting this is a common convention when concatenating $\sum$ terms, that they are nested with implied parens as you show. I'm sure there will be some discipline that does not do that, but all the ML I have read does this.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-18T09:35:13.600Z
Answers.Text: I'm sorry, I'm coming back and got confused, shouldn't be like this:
  $$v_{\pi}(s) =\sum_a \left( \pi(a \mid s) \sum_{s',r} p(s',r \mid s,a)[r+ \gamma v_\pi(s')] \right)$$
  Since the second sum uses the same $a$ index.. Putting the indexes the way @nbro does implies the sums are independent
Answers.UserDisplayName: nammerkage
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-18T11:47:51.640Z
Answers.Text: @nammerkage You can also write that **and** with the parentheses that I show in my answer. Both are ok. Parentheses here are used just to emphasize the precedence of the operations. So, you can also write $$v_{\pi}(s) =\sum_a \left( \pi(a \mid s) \left( \sum_{s',r} p(s',r \mid s,a)[r+ \gamma v_\pi(s')]\right) \right)$$
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-18T08:35:45.067Z
Answers.LastActivityDate: 2022-01-18T08:35:45.067Z
Answers.UserDisplayName: nbro
Body: <p>I'm not having a lot of intuition about the equation. I have this Bellman update rule:</p>
  <p><span class="math-container">$$v_{\pi}(s) =\sum_a \pi(a|s)\sum_{s',r} p(s',r|s,a)[r+ \gamma v_{k}(s')]$$</span></p>
  <p>But where are the parenthesis? Is the second sum using the index <span class="math-container">$a$</span> from the first sum? Or is it independent, and can I move out the <span class="math-container">$[r+ \gamma v_{k}(s')]$</span> term out of the sum?</p>
  
Tags: <reinforcement-learning><notation><bellman-equations>
Title: Where are the parentheses in the Bellman update rule?
UserDisplayName: nammerkage
ThreadText: # Where are the parentheses in the Bellman update rule?
  
  (Asked by: username_0 on 2022-01-18)
  
  I'm not having a lot of intuition about the equation. I have this Bellman update rule:
  
  $$v\_{\pi}(s) =\sum\_a \pi(a|s)\sum\_{s',r} p(s',r|s,a)[r+ \gamma v\_{k}(s')]$$
  
  But where are the parenthesis? Is the second sum using the index $a$ from the first sum? Or is it independent, and can I move out the $[r+ \gamma v\_{k}(s')]$ term out of the sum?
  
  **Tags:** reinforcement-learning, notation, bellman-equations
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-01-18. Score: 3 | Accepted answer
  
  Here's your equation with an additional couple of parenthesis that emphasizes the order of the operations (note that you had a small typo in your original equation).
  
  $$v\_{\pi}(s) =\sum\_a \pi(a \mid s) \left(\sum\_{s',r} p(s',r \mid s,a)[r+ \gamma v\_\pi(s')] \right)$$
  
  Now, let me answer your other questions.
  
  > Is the second sum using the index $a$ from the first sum?
  
  Yes.
  
  > Or is it independent, and can I move out the $[r+ \gamma v\_\pi(s')]$ term out of the sum?
  
  No, and you cannot move this term out of the sum because the second sum is a sum over $r$ and $s'$ and $r+ \gamma v\_\pi(s')$ depends on those terms.
  
  Note that $v\_{\pi}(s)$ is defined as an expectation and that $\pi(a \mid s)$ (the policy) and $p(s',r \mid s,a)$ (the [model](https://ai.stackexchange.com/a/8820/2444)) are probability distributions.
  
  > **Comments:**
  > **username_2**: Worth noting this is a common convention when concatenating $\sum$ terms, that they are nested with implied parens as you show. I'm sure there will be some discipline that does not do that, but all the ML I have read does this.  
  > **username_0**: I'm sorry, I'm coming back and got confused, shouldn't be like this: $$v_{\pi}(s) =\sum_a \left( \pi(a \mid s) \sum_{s',r} p(s',r \mid s,a)[r+ \gamma v_\pi(s')] \right)$$ Since the second sum uses the same $a$ index.. Putting the indexes the way @username_1 does implies the sums are independent  
  > **username_1**: @username_0 You can also write that **and** with the parentheses that I show in my answer. Both are ok. Parentheses here are used just to emphasize the precedence of the operations. So, you can also write $$v_{\pi}(s) =\sum_a \left( \pi(a \mid s) \left( \sum_{s',r} p(s',r \mid s,a)[r+ \gamma v_\pi(s')]\right) \right)$$  
  
  ---
  
  