Answers.Body: <p>Having low variance is important in general as it reduces the number of samples needed to obtain accurate estimates. This is the case for all statistical machine learning, not just reinforcement learning.</p>
  
  <p>In general, if you are estimating a mean or expected quantity by taking many samples, the variation in the error is proportional to <span class="math-container">$\frac{\sigma}{\sqrt{N}}$</span> for a direct arithemtic mean of all samples, and behaves similarly for other averaging approaches (such as recency-weighted means using a learning rate). The bounds on accuracy can be made better by either increasing <span class="math-container">$N$</span> i.e. taking more samples, or by decreasing the variance <span class="math-container">$\sigma^2$</span>.</p>
  
  <p>So anything you can do to reduce variance in your measurements has a direct consequence of reducing the number of samples required to achieve the same degree of accuracy.</p>
  
  <p>In the case of off-policy reinforcement learning, there is added variance - compared to on-policy learning - due to different probabilities of taking an action in behaviour and target policies. This is due to the need to adjust reward signals using importance sampling - multiplying by the importance sampling ratio will make the reward signal vary more (in fact it can become unbounded). This is not really any more of a challenge than any other source of variance, but as it interferes with the goal of speedy learning, a lot of research effort has been put into methods that reduce the variance.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-29T20:30:57.057Z
Answers.LastActivityDate: 2019-09-29T20:30:57.057Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>Bias is not necessarily bad, even though the term <em>bias</em> usually has a negative connotation. In fact, in machine learning, <a href="https://en.wikipedia.org/wiki/Inductive_bias" rel="nofollow noreferrer">inductive bias</a> is quite important and necessary. For example, if you want to learn a function <span class="math-container">$f(x) = y$</span>, where <span class="math-container">$x \in \mathcal{X}$</span> and <span class="math-container">$y \in \mathcal{Y}$</span>, you often just have a <em>finite</em> dataset <span class="math-container">$\mathcal{D} = \{ (x_i, y_i)\}_{i=1}^N$</span>, which may not contain all possible <span class="math-container">$(x, y)$</span> pairs associated with <span class="math-container">$f$</span>. In that case, <span class="math-container">$\mathcal{D}$</span> may not contain enough information to learn <span class="math-container">$f$</span>, so you need to assume that <span class="math-container">$f$</span> behaves in a certain way or that the input and output spaces have certain characteristics. A typical way of dealing with finite datasets is to introduce noise during the learning process (which is a <em>regularization technique</em>). </p>
  
  <p>However, bias can lead to sub-optimal solutions. For example, you could assume that <span class="math-container">$f$</span> is a lot more complex than the function <span class="math-container">$\hat{f}$</span> that maps <span class="math-container">$x_i$</span> to <span class="math-container">$y_i$</span> (of <span class="math-container">$\mathcal{D}$</span>), for <span class="math-container">$i=1, \dots, N$</span>. So, to solve this issue, you could introduce a lot of noise, while, in reality, <span class="math-container">$\hat{f}$</span> may be extremely similar to <span class="math-container">$f$</span>, even though not exactly the same, so, in reality, you may not need all this noise.</p>
  
  <p>Why is low variance desirable? Essentially, while you are learning something, it is easier to learn regular patterns as opposed to more irregular ones. For example, <span class="math-container">$1, 2, 1, 2, 1, 2$</span> is a relatively regular sequence compared to <span class="math-container">$8, 2, 5, 6, 1, 7, 99$</span>, which is thus harder to learn (or memorise) than the former.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-29T21:36:13.720Z
Answers.Text: This is an addition to [Neil's answer](https://ai.stackexchange.com/a/15664/2444).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-29T21:27:08.230Z
Answers.LastActivityDate: 2019-09-29T21:39:14.717Z
Answers.LastEditDate: 2019-09-29T21:39:14.717Z
Answers.UserDisplayName: nbro
Body: <p>Intuitively, I understand that having an unbiased estimate of a policy is important because being biased just means that our estimate is distant from the truth value. </p>
  
  <p>However, I don't understand clearly why having lower variance is important. Is that because, in offline policy evaluation, we can have only 'one' estimate with a stream of data, and we don't know if it is because of variance or bias when our estimate is far from the truth value? Basically, variance acts like bias. </p>
  
  <p>Also, if that is the case, why having variance is preferable to having a bias?</p>
  
Tags: <reinforcement-learning><policies><bias-variance-tradeoff>
Title: Why is having low variance important in offline policy evaluation of reinforcement learning?
UserDisplayName: Hunnam 
ThreadText: # Why is having low variance important in offline policy evaluation of reinforcement learning?
  
  (Asked by: username_0 on 2019-09-29)
  
  Intuitively, I understand that having an unbiased estimate of a policy is important because being biased just means that our estimate is distant from the truth value.
  
  However, I don't understand clearly why having lower variance is important. Is that because, in offline policy evaluation, we can have only 'one' estimate with a stream of data, and we don't know if it is because of variance or bias when our estimate is far from the truth value? Basically, variance acts like bias.
  
  Also, if that is the case, why having variance is preferable to having a bias?
  
  **Tags:** reinforcement-learning, policies, bias-variance-tradeoff
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-09-29. Score: 1 | Accepted answer
  
  Having low variance is important in general as it reduces the number of samples needed to obtain accurate estimates. This is the case for all statistical machine learning, not just reinforcement learning.
  
  In general, if you are estimating a mean or expected quantity by taking many samples, the variation in the error is proportional to $\frac{\sigma}{\sqrt{N}}$ for a direct arithemtic mean of all samples, and behaves similarly for other averaging approaches (such as recency-weighted means using a learning rate). The bounds on accuracy can be made better by either increasing $N$ i.e. taking more samples, or by decreasing the variance $\sigma^2$.
  
  So anything you can do to reduce variance in your measurements has a direct consequence of reducing the number of samples required to achieve the same degree of accuracy.
  
  In the case of off-policy reinforcement learning, there is added variance - compared to on-policy learning - due to different probabilities of taking an action in behaviour and target policies. This is due to the need to adjust reward signals using importance sampling - multiplying by the importance sampling ratio will make the reward signal vary more (in fact it can become unbounded). This is not really any more of a challenge than any other source of variance, but as it interferes with the goal of speedy learning, a lot of research effort has been put into methods that reduce the variance.
  
  
  ---
  
  ### Answer by username_2 on 2019-09-29. Score: 1
  
  Bias is not necessarily bad, even though the term *bias* usually has a negative connotation. In fact, in machine learning, [inductive bias](https://en.wikipedia.org/wiki/Inductive_bias) is quite important and necessary. For example, if you want to learn a function $f(x) = y$, where $x \in \mathcal{X}$ and $y \in \mathcal{Y}$, you often just have a *finite* dataset $\mathcal{D} = \{ (x\_i, y\_i)\}\_{i=1}^N$, which may not contain all possible $(x, y)$ pairs associated with $f$. In that case, $\mathcal{D}$ may not contain enough information to learn $f$, so you need to assume that $f$ behaves in a certain way or that the input and output spaces have certain characteristics. A typical way of dealing with finite datasets is to introduce noise during the learning process (which is a *regularization technique*).
  
  However, bias can lead to sub-optimal solutions. For example, you could assume that $f$ is a lot more complex than the function $\hat{f}$ that maps $x\_i$ to $y\_i$ (of $\mathcal{D}$), for $i=1, \dots, N$. So, to solve this issue, you could introduce a lot of noise, while, in reality, $\hat{f}$ may be extremely similar to $f$, even though not exactly the same, so, in reality, you may not need all this noise.
  
  Why is low variance desirable? Essentially, while you are learning something, it is easier to learn regular patterns as opposed to more irregular ones. For example, $1, 2, 1, 2, 1, 2$ is a relatively regular sequence compared to $8, 2, 5, 6, 1, 7, 99$, which is thus harder to learn (or memorise) than the former.
  
  > **Comments:**
  > **username_2**: This is an addition to [Neil's answer](https://ai.stackexchange.com/a/15664/2444).  
  
  ---
  
  