Answers.Body: <p>Sutton and Barto are being a bit loose with notation here.</p>
  <p>If you had</p>
  <ul>
  <li>a random variable  <span class="math-container">$\mathbf{X}$</span>, with individual observable values <span class="math-container">$x$</span> drawn from set <span class="math-container">$\mathcal{X}$</span></li>
  <li>a discrete probability function for <span class="math-container">$\mathbf{X}$</span>: <span class="math-container">$p(x)$</span> where <span class="math-container">$\sum_{x \in \mathcal{X}} p(x) = 1$</span></li>
  <li>and a function <span class="math-container">$f(x): \mathcal{X} \rightarrow \mathcal{Y}$</span> i.e that maps any <span class="math-container">$x$</span> to another type</li>
  </ul>
  <p>Then you can process <span class="math-container">$f(\mathbf{X})$</span>, the same function applied to all possible <span class="math-container">$x$</span>. This returns the distribution for a new random variable <span class="math-container">$\mathbf{Y}$</span> - all the output values of <span class="math-container">$f(x)$</span> - with the probability of each value depending on <span class="math-container">$p(x)$</span></p>
  <p>In Sutton &amp; Barto, they are treating <span class="math-container">$\pi'(s)$</span> as a random variable for the action choice, like <span class="math-container">$\mathbf{X}$</span> (this is where they are being loose, it's not the deterministic policy). Whilst <span class="math-container">$\pi'(a|s)$</span> is equivalent to <span class="math-container">$p(x)$</span> in the above description.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-24T14:59:49.057Z
Answers.Text: In your first item, you wrote three different versions of the letter X. How do you pronounce them to differentiate them from each other?
Answers.UserDisplayName: evaristegd
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-24T15:03:09.987Z
Answers.Text: @evaristegd There is an observable value, a set, and a distribution for the same variable. So you would say "the observed value x", or "the set of all possible x", or "the probability distribution for observing specific x" etc. For concise notation, it makes sense to use some variants of the same letter to show that they are related. I used the font variants that are typically used in S&B.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-24T18:53:38.387Z
Answers.Text: i would also say that makes no sense to define Q using a distribution as action, because that’s literally the definition of $V^\pi(s)$…
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-24T19:37:01.383Z
Answers.Text: @Alberto except that is the point of the proof for Policy Improvement Theorem over epsilon-greedy policies, that this is from. It's showing it in a similar way to the deterministic policy version.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-24T19:40:59.043Z
Answers.Text: @Alberto - also, the value that results is neither $V^{\pi}(s)$ nor $V^{\pi'}(s)$, but something in-between. The proof eventually rolls out the change of policy over the whole trajectory, transforming $V^{\pi}(s)$ to $V^{\pi'}(s)$ and demonstrating $V^{\pi}(s) \leq V^{\pi'}(s)$
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-24T21:16:48.790Z
Answers.Text: @NeilSlater sorry, it didn't wanted to sound like a critique of your answer, but just of the notation used in general
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-24T21:25:25.870Z
Answers.Text: @NeilSlater, what do you mean by "distribution "?  What is the difference between distribution  and p(x)? I think by "distribution "  you mean a random variable "", right?
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-24T21:28:34.797Z
Answers.Text: @DSPinfinity Yes, random variable is what Sutton&Barto call it, and probably the better term. However, we do want to make the distribution of it explictly known.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-24T21:31:15.637Z
Answers.Text: @NeilSlater  Then, could you please update your answer by using the term random variable  for "distribution "?
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-24T21:32:56.010Z
Answers.Text: @DSPinfinity done
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-25T01:00:53.867Z
Answers.Text: @NeilSlater, I think still your explanation  needs an update to make the connection between  $X$, $f(X)$ and $Y$ and the  stated problem variables and/or expressions. I.e., what are  $X$, $f(X)$,  $Y$  in the stated problem? It will be perfect if in your answer you make this correspondence clearer.
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-24T13:57:05.730Z
Answers.LastActivityDate: 2024-03-24T21:32:45.360Z
Answers.LastEditDate: 2024-03-24T21:32:45.360Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>LHS <span class="math-container">$\pi’(s)$</span> is just <span class="math-container">$\pi’(.|s)$</span> which denotes the whole probability distribution of the said policy given state <span class="math-container">$s$</span>, while your RHS <span class="math-container">$\pi’(a|s)$</span> is just a specific conditional probability when the random variable A=a, given the same state <span class="math-container">$s$</span>. Once you understand this, you can also immediately see the reason for the first line of your equation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-24T19:02:59.500Z
Answers.LastActivityDate: 2024-03-24T19:02:59.500Z
Answers.UserDisplayName: cinch
Body: <p>Sutton-Barto, page 101, Eq (5.2):
  Assume that <span class="math-container">$\pi^\prime$</span> is the <span class="math-container">$\epsilon$</span>-greedy policy. Then,</p>
  <p><span class="math-container">\begin{align}
   q_{\pi}\big(s,\pi'(s)\big)&amp;= \sum_{a}\pi'(a|s)q_{\pi}(s,a) \\
                             &amp;= \frac{\varepsilon}{|\mathcal{A}(s)|}\sum_{a}q_{\pi}(s,a)+(1-\varepsilon)\max_{a}q_{\pi}(s,a)\\
  &amp; \cdots
  \end{align}</span></p>
  <p>I am confused by the first line in first equation. My confusion points are as follows.</p>
  <p>(1) On the left we have <span class="math-container">$\pi'(s)$</span> but on the right <span class="math-container">$\pi'(a|s)$</span>. Since <span class="math-container">$\pi^\prime$</span> is stochastic, how can one write <span class="math-container">$\pi'(s)$</span>?</p>
  <p>(2) Why does the first line in equation follow?</p>
  
Tags: <reinforcement-learning><sutton-barto>
Title: Confusing point in Sutton-Barto: replacing $a$ in $q(s,a)$ with a stochastic policy $\pi^\prime$
UserDisplayName: DSPinfinity
ThreadText: # Confusing point in Sutton-Barto: replacing $a$ in $q(s,a)$ with a stochastic policy $\pi^\prime$
  
  (Asked by: username_0 on 2024-03-24)
  
  Sutton-Barto, page 101, Eq (5.2):
  Assume that $\pi^\prime$ is the $\epsilon$-greedy policy. Then,
  
  \begin{align}
  q\_{\pi}\big(s,\pi'(s)\big)&= \sum\_{a}\pi'(a|s)q\_{\pi}(s,a) \\
  &= \frac{\varepsilon}{|\mathcal{A}(s)|}\sum\_{a}q\_{\pi}(s,a)+(1-\varepsilon)\max\_{a}q\_{\pi}(s,a)\\
  & \cdots
  \end{align}
  
  I am confused by the first line in first equation. My confusion points are as follows.
  
  (1) On the left we have $\pi'(s)$ but on the right $\pi'(a|s)$. Since $\pi^\prime$ is stochastic, how can one write $\pi'(s)$?
  
  (2) Why does the first line in equation follow?
  
  **Tags:** reinforcement-learning, sutton-barto
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-03-24. Score: 3
  
  Sutton and Barto are being a bit loose with notation here.
  
  If you had
  
  * a random variable $\mathbf{X}$, with individual observable values $x$ drawn from set $\mathcal{X}$
  * a discrete probability function for $\mathbf{X}$: $p(x)$ where $\sum\_{x \in \mathcal{X}} p(x) = 1$
  * and a function $f(x): \mathcal{X} \rightarrow \mathcal{Y}$ i.e that maps any $x$ to another type
  
  Then you can process $f(\mathbf{X})$, the same function applied to all possible $x$. This returns the distribution for a new random variable $\mathbf{Y}$ - all the output values of $f(x)$ - with the probability of each value depending on $p(x)$
  
  In Sutton & Barto, they are treating $\pi'(s)$ as a random variable for the action choice, like $\mathbf{X}$ (this is where they are being loose, it's not the deterministic policy). Whilst $\pi'(a|s)$ is equivalent to $p(x)$ in the above description.
  
  > **Comments:**
  > **username_2**: In your first item, you wrote three different versions of the letter X. How do you pronounce them to differentiate them from each other?  
  > **username_1**: @username_2 There is an observable value, a set, and a distribution for the same variable. So you would say "the observed value x", or "the set of all possible x", or "the probability distribution for observing specific x" etc. For concise notation, it makes sense to use some variants of the same letter to show that they are related. I used the font variants that are typically used in S&B.  
  > **username_3**: i would also say that makes no sense to define Q using a distribution as action, because that’s literally the definition of $V^\pi(s)$…  
  > **username_1**: @username_3 except that is the point of the proof for Policy Improvement Theorem over epsilon-greedy policies, that this is from. It's showing it in a similar way to the deterministic policy version.  
  > **username_1**: @username_3 - also, the value that results is neither $V^{\pi}(s)$ nor $V^{\pi'}(s)$, but something in-between. The proof eventually rolls out the change of policy over the whole trajectory, transforming $V^{\pi}(s)$ to $V^{\pi'}(s)$ and demonstrating $V^{\pi}(s) \leq V^{\pi'}(s)$  
  > **username_3**: @username_1 sorry, it didn't wanted to sound like a critique of your answer, but just of the notation used in general  
  > **username_0**: @username_1, what do you mean by "distribution "?  What is the difference between distribution  and p(x)? I think by "distribution "  you mean a random variable "", right?  
  > **username_1**: @username_0 Yes, random variable is what Sutton&Barto call it, and probably the better term. However, we do want to make the distribution of it explictly known.  
  > **username_0**: @username_1  Then, could you please update your answer by using the term random variable  for "distribution "?  
  > **username_1**: @username_0 done  
  > **username_0**: @username_1, I think still your explanation  needs an update to make the connection between  $X$, $f(X)$ and $Y$ and the  stated problem variables and/or expressions. I.e., what are  $X$, $f(X)$,  $Y$  in the stated problem? It will be perfect if in your answer you make this correspondence clearer.  
  
  ---
  
  ### Answer by username_4 on 2024-03-24. Score: 2
  
  LHS $\pi’(s)$ is just $\pi’(.|s)$ which denotes the whole probability distribution of the said policy given state $s$, while your RHS $\pi’(a|s)$ is just a specific conditional probability when the random variable A=a, given the same state $s$. Once you understand this, you can also immediately see the reason for the first line of your equation.
  
  
  ---
  
  