Answers.Body: <blockquote>
  <p>why we calculate expected return for policy <span class="math-container">$b$</span> instead of <span class="math-container">$\pi$</span>?</p>
  </blockquote>
  <p>The expected return for policy <span class="math-container">$b$</span> is not being calculated here:</p>
  <p><span class="math-container">\begin{align}
  \mathbb{E}_b \Bigg[(\prod_{t=0}^{T-1} \frac{\pi(A_t|S_t)}{b(A_t|S_t)}G_0 )^2\Bigg]
  \end{align}</span></p>
  <p>This expression is part of the variance of the importance-sampled expected return for <span class="math-container">$\pi$</span>, given that your observations of <span class="math-container">$G_0$</span> are all made following <span class="math-container">$b$</span>. The expectation of this value is shown as expected following policy <span class="math-container">$b$</span>, because that is what you directly observe.</p>
  <p>If you were instead observing returns from following the policy <span class="math-container">$\pi$</span>, you wouldn't have the problem with possible unbound variance when trying to convert it.</p>
  <p>The proof is showing a problem inherent to the conversion maths. The importance sampling <strong>is</strong> still approximating <span class="math-container">$\mathbb{E}_{\pi} \Big[G_0\Big]$</span>, but the proof needs to look at the conversion in detail to show the possibility of infinite variance.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-08T16:44:17.963Z
Answers.LastActivityDate: 2024-01-08T16:51:43.377Z
Answers.LastEditDate: 2024-01-08T16:51:43.377Z
Answers.UserDisplayName: Neil Slater
Body: <p>From &quot;Reinforcement Learning: An introduction (2nd ed.)&quot; by Richard S. Sutton and Andrew G. Barto, on page 107 and 108:</p>
  <blockquote>
  <p>We can verify that the variance of the importance-sampling scaled
  returns is infinite in this last example. The variance of a random
  variable X  is the expected value of the deviation from its mean X¯:</p>
  </blockquote>
  <p><span class="math-container">\begin{align}
  \mathrm{Var}[X] \doteq \mathbb{E} \Big[ (X - \bar{X})^2\Big] = \mathbb{E} \Big[ X^2 - 2X\bar{X} + \bar{X}^2 \Big] = \mathbb{E} \Big[ X^2\Big] - \bar{X}^2
  \end{align}</span></p>
  <blockquote>
  <p>If the mean of the random variable is finite (as in the example),
  having an infinite variance means that the expectation of the square
  of the random variable is infinite. Thus, we need only show that the
  expected square of the importance-sampling scaled return is infinite:</p>
  </blockquote>
  <p><span class="math-container">\begin{align}
  \mathbb{E}\_b \Bigg[(\prod_{t=0}^{T-1} \frac{\pi(A_t|S_t)}{b(A_t|S_t)}G_0 )^2\Bigg]
  \end{align}</span></p>
  <p>Question is following: why we calculate expected return for policy <span class="math-container">$b$</span> instead of <span class="math-container">$\pi$</span>? If we approximate returns for <span class="math-container">$\pi$</span> using importance sampling aren't we supposed to calculate <span class="math-container">$\mathbb{E}\_\pi$</span>?</p>
  
Tags: <reinforcement-learning><sutton-barto>
Title: Sutton & Barto: Why expected square of the importance-sampling-scaled return is for policy b?
UserDisplayName: Max Gorbunov
ThreadText: # Sutton & Barto: Why expected square of the importance-sampling-scaled return is for policy b?
  
  (Asked by: username_0 on 2024-01-08)
  
  From "Reinforcement Learning: An introduction (2nd ed.)" by Richard S. Sutton and Andrew G. Barto, on page 107 and 108:
  
  > We can verify that the variance of the importance-sampling scaled
  > returns is infinite in this last example. The variance of a random
  > variable X is the expected value of the deviation from its mean X¯:
  
  \begin{align}
  \mathrm{Var}[X] \doteq \mathbb{E} \Big[ (X - \bar{X})^2\Big] = \mathbb{E} \Big[ X^2 - 2X\bar{X} + \bar{X}^2 \Big] = \mathbb{E} \Big[ X^2\Big] - \bar{X}^2
  \end{align}
  
  > If the mean of the random variable is finite (as in the example),
  > having an infinite variance means that the expectation of the square
  > of the random variable is infinite. Thus, we need only show that the
  > expected square of the importance-sampling scaled return is infinite:
  
  \begin{align}
  \mathbb{E}\\_b \Bigg[(\prod\_{t=0}^{T-1} \frac{\pi(A\_t|S\_t)}{b(A\_t|S\_t)}G\_0 )^2\Bigg]
  \end{align}
  
  Question is following: why we calculate expected return for policy $b$ instead of $\pi$? If we approximate returns for $\pi$ using importance sampling aren't we supposed to calculate $\mathbb{E}\\_\pi$?
  
  **Tags:** reinforcement-learning, sutton-barto
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-01-08. Score: 2 | Accepted answer
  
  > why we calculate expected return for policy $b$ instead of $\pi$?
  
  The expected return for policy $b$ is not being calculated here:
  
  \begin{align}
  \mathbb{E}\_b \Bigg[(\prod\_{t=0}^{T-1} \frac{\pi(A\_t|S\_t)}{b(A\_t|S\_t)}G\_0 )^2\Bigg]
  \end{align}
  
  This expression is part of the variance of the importance-sampled expected return for $\pi$, given that your observations of $G\_0$ are all made following $b$. The expectation of this value is shown as expected following policy $b$, because that is what you directly observe.
  
  If you were instead observing returns from following the policy $\pi$, you wouldn't have the problem with possible unbound variance when trying to convert it.
  
  The proof is showing a problem inherent to the conversion maths. The importance sampling **is** still approximating $\mathbb{E}\_{\pi} \Big[G\_0\Big]$, but the proof needs to look at the conversion in detail to show the possibility of infinite variance.
  
  
  ---
  
  