Answers.Body: <p>Recall that the definition of a value function is
  <span class="math-container">$$v_\pi(s) = \mathbb{E}\left[G_t | S_t = s\right]\;.$$</span>
  That is, the expected future returns given from state <span class="math-container">$s$</span> at time <span class="math-container">$t$</span> when we follow our policy <span class="math-container">$\pi$</span> -- i.e. our trajectory is generated according to <span class="math-container">$\pi$</span>.</p>
  <p>Using Monte Carlo methods we typically will estimate our value function by looking at the empirical mean of rewards we see throughout many training episodes, i.e. we will generate many episodes, keep track of all the rewards we see from state <span class="math-container">$s$</span> onwards across all of our episodes (this may be the first visit method or the all visit methods) and use these to approximate the expectation that is our value function.</p>
  <p>The key here is that to approximate the value function in this way, then the episodes must be generated according to our policy <span class="math-container">$\pi$</span>. If we choose the actions in an episode according to some other policy <span class="math-container">$\beta$</span> then we cannot use these episodes to approximate the expectation directly. As an example, this would be like trying to approximate the mean of a Normal(0, 1) distribution with data drawn from a Normal(10, 1) distribution.</p>
  <p>To account for the fact that the actions came from a different distribution, we have to reweight the returns according to an importance sampling ratio. To see why we need importance sampling, <a href="https://ai.stackexchange.com/questions/25553/why-do-we-need-importance-sampling/25559#25559">see this question/answer.</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-12T10:37:09.010Z
Answers.Text: Ok. What we re-weight are the returns, so the values of each state value or state value action. Thank you @David Ireland
Answers.UserDisplayName: Hermes Morales
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-11T20:19:00.057Z
Answers.LastActivityDate: 2021-04-11T20:19:00.057Z
Answers.UserDisplayName: David
Answers.Body: <blockquote>
  <p>We estimate a value using sampling on whole episodes, and we take this values to construct the target policy.</p>
  </blockquote>
  <p>The crucial bit that you are missing is that there is no single value of <span class="math-container">$V(s)$</span> (or <span class="math-container">$Q(s,a)$</span>) of a state (or a state action pair). These value functions are always defined with respect to some policy <span class="math-container">$\pi(a|s)$</span> and is given the notation of <span class="math-container">$V^{\pi}(s)$</span> (or <span class="math-container">$Q^{\pi}(s,a)$</span>).</p>
  <p>The off-policy learning problems are arising when you have two policies: the generation policy <span class="math-container">$\mu(a|s)$</span> and the target policy <span class="math-container">$\pi(a|s)$</span>. Your MC sampling data came from an agent following <span class="math-container">$\mu$</span>, while you want to improve your target policy <span class="math-container">$\pi$</span>. It is pretty straightforward from here that you'd need to weight your calculations with factors like <span class="math-container">$\frac{\pi(a_i|s_i)}{\mu(a_i|s_i)}$</span> - that's what importance sampling is.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-11T13:14:58.550Z
Answers.LastActivityDate: 2021-04-11T22:17:13.110Z
Answers.LastEditDate: 2021-04-11T22:17:13.110Z
Answers.UserDisplayName: Kostya
Body: <p>I do not understand the link of importance sampling to Monte Carlo off-policy learning.</p>
  <p>We estimate a value using sampling on whole episodes, and we take these values to construct the target policy.</p>
  <p>So, it is possible that in the target policy, we could have state values (or state action values) coming from different trajectories.</p>
  <p>If the above is true, and if the values depend on the subsequent actions (the behavior policy), there is something wrong there, or else, better, something I do not understand.</p>
  <p>Linking this question with importance sampling, do we use this ro value to correct this inconsistency?</p>
  <p>Any clarification is welcome.</p>
  
Tags: <reinforcement-learning><monte-carlo-methods><off-policy-methods><importance-sampling>
Title: With Monte Carlo off-policy learning what do we correct by using importance sampling?
UserDisplayName: Hermes Morales
ThreadText: # With Monte Carlo off-policy learning what do we correct by using importance sampling?
  
  (Asked by: username_0 on 2021-04-11)
  
  I do not understand the link of importance sampling to Monte Carlo off-policy learning.
  
  We estimate a value using sampling on whole episodes, and we take these values to construct the target policy.
  
  So, it is possible that in the target policy, we could have state values (or state action values) coming from different trajectories.
  
  If the above is true, and if the values depend on the subsequent actions (the behavior policy), there is something wrong there, or else, better, something I do not understand.
  
  Linking this question with importance sampling, do we use this ro value to correct this inconsistency?
  
  Any clarification is welcome.
  
  **Tags:** reinforcement-learning, monte-carlo-methods, off-policy-methods, importance-sampling
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-04-11. Score: 0 | Accepted answer
  
  Recall that the definition of a value function is
  $$v\_\pi(s) = \mathbb{E}\left[G\_t | S\_t = s\right]\;.$$
  That is, the expected future returns given from state $s$ at time $t$ when we follow our policy $\pi$ -- i.e. our trajectory is generated according to $\pi$.
  
  Using Monte Carlo methods we typically will estimate our value function by looking at the empirical mean of rewards we see throughout many training episodes, i.e. we will generate many episodes, keep track of all the rewards we see from state $s$ onwards across all of our episodes (this may be the first visit method or the all visit methods) and use these to approximate the expectation that is our value function.
  
  The key here is that to approximate the value function in this way, then the episodes must be generated according to our policy $\pi$. If we choose the actions in an episode according to some other policy $\beta$ then we cannot use these episodes to approximate the expectation directly. As an example, this would be like trying to approximate the mean of a Normal(0, 1) distribution with data drawn from a Normal(10, 1) distribution.
  
  To account for the fact that the actions came from a different distribution, we have to reweight the returns according to an importance sampling ratio. To see why we need importance sampling, [see this question/answer.](https://ai.stackexchange.com/questions/25553/why-do-we-need-importance-sampling/25559#25559)
  
  > **Comments:**
  > **username_0**: Ok. What we re-weight are the returns, so the values of each state value or state value action. Thank you @username_1 Ireland  
  
  ---
  
  ### Answer by username_2 on 2021-04-11. Score: 3
  
  > We estimate a value using sampling on whole episodes, and we take this values to construct the target policy.
  
  The crucial bit that you are missing is that there is no single value of $V(s)$ (or $Q(s,a)$) of a state (or a state action pair). These value functions are always defined with respect to some policy $\pi(a|s)$ and is given the notation of $V^{\pi}(s)$ (or $Q^{\pi}(s,a)$).
  
  The off-policy learning problems are arising when you have two policies: the generation policy $\mu(a|s)$ and the target policy $\pi(a|s)$. Your MC sampling data came from an agent following $\mu$, while you want to improve your target policy $\pi$. It is pretty straightforward from here that you'd need to weight your calculations with factors like $\frac{\pi(a\_i|s\_i)}{\mu(a\_i|s\_i)}$ - that's what importance sampling is.
  
  
  ---
  
  