Answers.Body: <p>What this means is that if you assume every test has some noise, if you only look at the tests that turned out well, some of the reason they turned out well was likely that noise was in their favor. As such, you expect regression to the mean if you were to re-test only the passing patches. They would appear less good, because  &quot;passing test&quot; is a biased sample.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-24T06:44:10.953Z
Answers.LastActivityDate: 2020-07-24T06:44:10.953Z
Answers.UserDisplayName: Oscar Smith
Body: <p><a href="https://github.com/glinscott/fishtest/wiki/Fishtest-faq#why-is-the-regression-test-bad" rel="nofollow noreferrer">Source</a></p>
  <p>Point 5:</p>
  <blockquote>
  <p>Elo estimates of single patches (SPRT runs) typically come with large error bars. Take this into account when adding Elo estimates. <strong>Furthermore, Elo estimates of passing patches are biased. The SPRT Elo estimates are only unbiased if one takes all patches into account, both passed and non-passed ones.</strong> As a result, the Elo gain measured by a regression test will typically be less than the sum of the estimated Elo gains of the individual patches since the previous regression test.</p>
  </blockquote>
  <p>I don't understand the highlighted sentences. Can someone explain?</p>
  
Tags: <stockfish>
Title: What does this paragraph about Stockfish's regression tests mean?
ThreadText: # What does this paragraph about Stockfish's regression tests mean?
  
  (Asked by: username_0 on 2020-07-24)
  
  [Source](https://github.com/glinscott/fishtest/wiki/Fishtest-faq#why-is-the-regression-test-bad)
  
  Point 5:
  
  > Elo estimates of single patches (SPRT runs) typically come with large error bars. Take this into account when adding Elo estimates. **Furthermore, Elo estimates of passing patches are biased. The SPRT Elo estimates are only unbiased if one takes all patches into account, both passed and non-passed ones.** As a result, the Elo gain measured by a regression test will typically be less than the sum of the estimated Elo gains of the individual patches since the previous regression test.
  
  I don't understand the highlighted sentences. Can someone explain?
  
  **Tags:** stockfish
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-07-24. Score: 7 | Accepted answer
  
  What this means is that if you assume every test has some noise, if you only look at the tests that turned out well, some of the reason they turned out well was likely that noise was in their favor. As such, you expect regression to the mean if you were to re-test only the passing patches. They would appear less good, because "passing test" is a biased sample.
  
  
  ---
  
  