Answers.Body: <p>The book presents a slight abuse of notation where <span class="math-container">$\pi'(s)$</span> is shorthand for a distribution of action values described by the more correct <span class="math-container">$\pi'(a|s)$</span>. At that point there is an implied function composition of <span class="math-container">$q_{\pi}$</span> over this distribution, resolved on the right hand side to its expectation.</p>
  <p>I believe it does this so that it can briefly show something familiar from the deterministic policy improvement theorem. You could almost read it as &quot;the equivalent of this term (LHS) taken from the previous proof must now be written like this (RHS)&quot;.</p>
  <p>It would be more correct notation to write something like this:</p>
  <p><span class="math-container">$$\mathbb{E}[q_{\pi}(s,A')|A' \sim \pi'] =  ...$$</span></p>
  <p>or perhaps just start with the right hand side.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-13T17:15:40.930Z
Answers.Text: @ Neil Slater,  strangely authors also  set $v_{\pi'}(s)=q_{\pi}(s,\pi'(s))$.  I cannot see why this holds.
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-13T17:27:40.377Z
Answers.Text: @DSPinfinity It holds provided the *only* change between $\pi$ and $\pi'$ is at the single state $s$, and that there are no cyclic paths that allow a revisit to $s$.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-13T17:33:20.870Z
Answers.Text: @ Neil Slater  $v_{\pi'}(s)$ means the value of state $s$ following $\pi'$,  $q_{\pi}(s, \pi'(s))$ means value of state state $s$  by applying $\pi'$  and then following $\pi$. So, how can they be equal? Am I missing something?
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-13T17:39:47.253Z
Answers.Text: They can be equal if the only difference between $\pi$ and $\pi'$ is localised to state $s$ (plus the no loops constraint), because $\pi$ and $\pi'$ are identical for all other states that could follow $s$.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-13T18:52:50.263Z
Answers.Text: @ Neil Slater, $\pi'$ is greedily updated from $\pi$ during a stage of policy improvement. However, during that stage it is not updated for the single state $s$ but for all encountered states of the episode. Hence, it seems that the difference is not only for the single state $s$. What do you think?
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-13T19:04:37.210Z
Answers.Text: In the case you describe, $v_{\pi'}(s) \neq q_{\pi}(s, \pi'(s))$ if $\pi' \neq \pi$. but I have not re-read the Sutton&Barto section to see how that relates to your comments. It is possible that in S&B, the constraints are implied or caveated correctly and we are discussing something slightly different
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-13T20:07:34.823Z
Answers.Text: @ Neil Slater, when you have time, I will be happy if you can read that part (page 101, Eq.5.2) and give me a feedback as an expert.
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-13T20:19:11.543Z
Answers.Text: @DSPinfinity I just re-read section 5.4 that includes that equation and related proof. I cannot see anywhere that it states $v_{\pi'}(s) = q_{\pi}(s, \pi'(s))$ nor anything similar.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-13T20:39:35.470Z
Answers.Text: @ Neil Slater,  at the beginning of page 102, he says "Thus, by the policy improvement theorem $\pi '\ge \pi$ (ie, $v_{\pi'}(s) \ge v_{\pi}(s) $" from which I concluded that he set $v_{\pi'}(s)=q_{\pi}(s,\pi'(s))$.
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-13T16:23:28.163Z
Answers.LastActivityDate: 2022-02-13T17:02:08.990Z
Answers.LastEditDate: 2022-02-13T17:02:08.990Z
Answers.UserDisplayName: Neil Slater
Body: <p>Let <span class="math-container">$\pi$</span> be an <span class="math-container">$\epsilon-soft$</span> policy with state-action value function <span class="math-container">$q_{\pi}(s,a)$</span>
  and <span class="math-container">$\pi'$</span> be an <span class="math-container">$\epsilon-greedy$</span> policy with respect to <span class="math-container">$q_{\pi}$</span>.</p>
  <p>In Sutton-Barto RL book  (page 101, eq. 5.2), they define</p>
  <p><span class="math-container">\begin{align}
  q_{\pi}(s, \pi'(s))=\displaystyle \sum_{a}\pi'(a|s)q_{\pi}(s,a).
  \end{align}</span>
  Normally <span class="math-container">$q_{\pi}(s, a)$</span> means taking action <span class="math-container">$a$</span>
  at state <span class="math-container">$s$</span> and then following the policy <span class="math-container">$\pi$</span>. Based on this convention, the notation
  <span class="math-container">$q_{\pi}(s, \pi'(s))$</span> is weird because <span class="math-container">$\pi'(s)$</span> is not a single action like <span class="math-container">$a$</span>.
  I.e., <span class="math-container">$\pi'$</span> is a stochastic policy
  and hence only <span class="math-container">$\pi'(a|s)$</span> makes sense.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-13T15:27:09.430Z
Comments.Text: Rather than writing "Confusion about..." in the title, can you put your **specific question** in the title?
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning>
Title: Confusion about state-action value function notation used in Sutton-Barto RL book
UserDisplayName: DSPinfinity
ThreadText: # Confusion about state-action value function notation used in Sutton-Barto RL book
  
  (Asked by: username_0 on 2022-02-13)
  
  Let $\pi$ be an $\epsilon-soft$ policy with state-action value function $q\_{\pi}(s,a)$
  and $\pi'$ be an $\epsilon-greedy$ policy with respect to $q\_{\pi}$.
  
  In Sutton-Barto RL book (page 101, eq. 5.2), they define
  
  \begin{align}
  q\_{\pi}(s, \pi'(s))=\displaystyle \sum\_{a}\pi'(a|s)q\_{\pi}(s,a).
  \end{align}
  Normally $q\_{\pi}(s, a)$ means taking action $a$
  at state $s$ and then following the policy $\pi$. Based on this convention, the notation
  $q\_{\pi}(s, \pi'(s))$ is weird because $\pi'(s)$ is not a single action like $a$.
  I.e., $\pi'$ is a stochastic policy
  and hence only $\pi'(a|s)$ makes sense.
  
  **Tags:** reinforcement-learning
  
  > **Comments:**
  > **username_1**: Rather than writing "Confusion about..." in the title, can you put your **specific question** in the title?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-02-13. Score: 2 | Accepted answer
  
  The book presents a slight abuse of notation where $\pi'(s)$ is shorthand for a distribution of action values described by the more correct $\pi'(a|s)$. At that point there is an implied function composition of $q\_{\pi}$ over this distribution, resolved on the right hand side to its expectation.
  
  I believe it does this so that it can briefly show something familiar from the deterministic policy improvement theorem. You could almost read it as "the equivalent of this term (LHS) taken from the previous proof must now be written like this (RHS)".
  
  It would be more correct notation to write something like this:
  
  $$\mathbb{E}[q\_{\pi}(s,A')|A' \sim \pi'] = ...$$
  
  or perhaps just start with the right hand side.
  
  > **Comments:**
  > **username_0**: @ username_2 username_2,  strangely authors also  set $v_{\pi'}(s)=q_{\pi}(s,\pi'(s))$.  I cannot see why this holds.  
  > **username_2**: @username_0 It holds provided the *only* change between $\pi$ and $\pi'$ is at the single state $s$, and that there are no cyclic paths that allow a revisit to $s$.  
  > **username_0**: @ username_2 username_2  $v_{\pi'}(s)$ means the value of state $s$ following $\pi'$,  $q_{\pi}(s, \pi'(s))$ means value of state state $s$  by applying $\pi'$  and then following $\pi$. So, how can they be equal? Am I missing something?  
  > **username_2**: They can be equal if the only difference between $\pi$ and $\pi'$ is localised to state $s$ (plus the no loops constraint), because $\pi$ and $\pi'$ are identical for all other states that could follow $s$.  
  > **username_0**: @ username_2 username_2, $\pi'$ is greedily updated from $\pi$ during a stage of policy improvement. However, during that stage it is not updated for the single state $s$ but for all encountered states of the episode. Hence, it seems that the difference is not only for the single state $s$. What do you think?  
  > **username_2**: In the case you describe, $v_{\pi'}(s) \neq q_{\pi}(s, \pi'(s))$ if $\pi' \neq \pi$. but I have not re-read the Sutton&Barto section to see how that relates to your comments. It is possible that in S&B, the constraints are implied or caveated correctly and we are discussing something slightly different  
  > **username_0**: @ username_2 username_2, when you have time, I will be happy if you can read that part (page 101, Eq.5.2) and give me a feedback as an expert.  
  > **username_2**: @username_0 I just re-read section 5.4 that includes that equation and related proof. I cannot see anywhere that it states $v_{\pi'}(s) = q_{\pi}(s, \pi'(s))$ nor anything similar.  
  > **username_0**: @ username_2 username_2,  at the beginning of page 102, he says "Thus, by the policy improvement theorem $\pi '\ge \pi$ (ie, $v_{\pi'}(s) \ge v_{\pi}(s) $" from which I concluded that he set $v_{\pi'}(s)=q_{\pi}(s,\pi'(s))$.  
  
  ---
  
  