Answers.Body: <p>Fishtest tests proposed changes to Stockfish's code with <a href="https://en.wikipedia.org/wiki/Sequential_probability_ratio_test" rel="nofollow noreferrer">SPRT</a> to determine if they should be accepted. Broadly speaking, there are two types of changes which are tested in this way (non-functional changes do not go through Fishtest): simplifications, and changes which intend to gain strength - elo changes.</p>
  
  <p>The crucial difference here is that simplifications (like the one referenced in the question) are tested for no regression, whereas elo changes are tested for an improvement: SPRT bounds of [0.5, 4.5] are used for elo changes and bounds of [-3, 1] are used for simplifications.</p>
  
  <p>The vast majority of changes to Stockfish recently have been simplifications, and as such they are not expected to increase strength.</p>
  
  <p>For example, the patch in the question is a simplification, and was tested with SPRT bounds [-3.00, 1.00]. The fact that it passed tells us that <em>there is a 95% chance that the change does not lose 3 elo</em>. If the test failed, we would know that <em>there is a 95% chance that the change does not gain 1 elo</em>. </p>
  
  <p>We can't say anything more precise than this, because SPRT stops immediately as soon as there is enough evidence to accept either hypothesis and doesn't attempt to measure the elo change more precisely. Although an elo difference of +2.18 is quoted at STC, all we know is that the lower error bound on the elo gain is <code>&gt;= -3.00</code>. </p>
  
  <p>There is insufficient evidence to say that the change produced an actual elo gain of +2.18, and thus claiming that this simplification gains 2 elo would be misleading.</p>
  
  <p>The small number of elo gaining changes is what results in Stockfish's slow progress at the moment.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-31T10:43:44.877Z
Answers.Text: Thanks for answer. Does this mean Stockfish developers classify their patches as "elo gaining" or "simplifications" before submitting them to fishtest?
Answers.UserDisplayName: Allure
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-31T10:59:57.410Z
Answers.Text: Stockfish has guidelines for testing methodology which describe the bounds that should be used for different types of changes. Developers submit a test to fishtest with settings that match one of the guidelines, unless they have a good reason to do otherwise. They can then open a pull request, quoting the results of the test.
Answers.UserDisplayName: konsolas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-31T10:13:17.973Z
Answers.LastActivityDate: 2019-05-31T10:13:17.973Z
Answers.UserDisplayName: konsolas
Body: <p><a href="https://nextchessmove.com/dev-builds" rel="nofollow noreferrer">NCM plays each Stockfish dev build 20,000 times against Stockfish 7. This yields an approximate Elo difference and establishes confidence in the strength of the dev builds.</a> As of time of writing, one can see from the plot that Stockfish's strength hasn't really budged for a month: version 20190324-1637 had a performance of +217.95 +/- 3.73 elo relative to Stockfish 7, while the latest version 20190420-0634 is +221.52 +/- 3.79 elo. The gap is within the error bars. </p>
  
  <p>(It's possible that this is sort of the expected performance increase in a month's time, but the latest version's results appear to be anomalously high. For example the next-latest version, 20190419-1533, is +218.61 +/- 3.77 and almost indistinguishable from the 20190324-1637 version.)</p>
  
  <p>The weird thing is, for each version of Stockfish to be promoted to the "main version", it has to beat the previous version. All these new versions of Stockfish (that aren't non-functional changes) have to do this. If we look at <a href="http://abrok.eu/stockfish/" rel="nofollow noreferrer">the catalogue of Stockfish patches</a>, we see patches such as:</p>
  
  <blockquote>
    <p>Author: protonspring 
    Date: Tue Apr 16 23:10:53 2019 +0200 
    Timestamp: 1555449053 </p>
    
    <p>Move pawnsOnSquares to Position (#2100) </p>
    
    <p>We can remove the values in Pawns if we just use the piece arrays in Position. This reduces the size of a pawn entry. This simplification passed individually, and in concert with ps_passedcount100 (removes passedCount storage in pawns.). </p>
    
    <p>STC 
    LLR: 2.95 (-2.94,2.94) [-3.00,1.00] 
    Total: 19957 W: 4529 L: 4404 D: 11024 Elo +2.18
    <a href="http://tests.stockfishchess.org/tests/view/5cb3c2d00ebc5925cf016f0d" rel="nofollow noreferrer">http://tests.stockfishchess.org/tests/view/5cb3c2d00ebc5925cf016f0d</a> </p>
    
    <p>Combo STC 
    LLR: 2.95 (-2.94,2.94) [-3.00,1.00] 
    Total: 17368 W: 3925 L: 3795 D: 9648 Elo +2.60
    <a href="http://tests.stockfishchess.org/tests/view/5cb3d3510ebc5925cf01709a" rel="nofollow noreferrer">http://tests.stockfishchess.org/tests/view/5cb3d3510ebc5925cf01709a</a> </p>
  </blockquote>
  
  <p>In other words this version of Stockfish defeated the previous one by ~2 elo. Why isn't this ~2 elo showing up in the NCM data? I know elo isn't transitive (i.e. if engine A beats engine B by 50 elo and engine B beats engine C by 50 elo, it doesn't mean engine A beats engine C by 100 elo), but this is just one patch, and there were plenty of elo-gaining patches that passed Fishtest. For NCM to show <em>no progress</em> for months, in spite of all these patches, doesn't make sense to me.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-24T05:27:16.333Z
Comments.Text: The patch you mention would appear to simply save some storage ( which could be useful for a lichess server playing thousands of opponents at once, for example ), but would not affect playing strength against a single opponent.
Comments.UserDisplayName: George Barwood
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-24T12:11:43.647Z
Comments.Text: @GeorgeBarwood isn't fishtest a 1v1 contest against a single opponent too? Also, what about the other patches on that website?
Comments.UserDisplayName: Allure
Tags: <engines><stockfish>
Title: Why does NCM show no progress for Stockfish when each new version has to beat the previous version?
ThreadText: # Why does NCM show no progress for Stockfish when each new version has to beat the previous version?
  
  (Asked by: username_0 on 2019-04-24)
  
  [NCM plays each Stockfish dev build 20,000 times against Stockfish 7. This yields an approximate Elo difference and establishes confidence in the strength of the dev builds.](https://nextchessmove.com/dev-builds) As of time of writing, one can see from the plot that Stockfish's strength hasn't really budged for a month: version 20190324-1637 had a performance of +217.95 +/- 3.73 elo relative to Stockfish 7, while the latest version 20190420-0634 is +221.52 +/- 3.79 elo. The gap is within the error bars.
  
  (It's possible that this is sort of the expected performance increase in a month's time, but the latest version's results appear to be anomalously high. For example the next-latest version, 20190419-1533, is +218.61 +/- 3.77 and almost indistinguishable from the 20190324-1637 version.)
  
  The weird thing is, for each version of Stockfish to be promoted to the "main version", it has to beat the previous version. All these new versions of Stockfish (that aren't non-functional changes) have to do this. If we look at [the catalogue of Stockfish patches](http://abrok.eu/stockfish/), we see patches such as:
  
  > Author: protonspring
  > Date: Tue Apr 16 23:10:53 2019 +0200
  > Timestamp: 1555449053
  >
  > Move pawnsOnSquares to Position (#2100)
  >
  > We can remove the values in Pawns if we just use the piece arrays in Position. This reduces the size of a pawn entry. This simplification passed individually, and in concert with ps\_passedcount100 (removes passedCount storage in pawns.).
  >
  > STC
  > LLR: 2.95 (-2.94,2.94) [-3.00,1.00]
  > Total: 19957 W: 4529 L: 4404 D: 11024 Elo +2.18
  > <http://tests.stockfishchess.org/tests/view/5cb3c2d00ebc5925cf016f0d>
  >
  > Combo STC
  > LLR: 2.95 (-2.94,2.94) [-3.00,1.00]
  > Total: 17368 W: 3925 L: 3795 D: 9648 Elo +2.60
  > <http://tests.stockfishchess.org/tests/view/5cb3d3510ebc5925cf01709a>
  
  In other words this version of Stockfish defeated the previous one by ~2 elo. Why isn't this ~2 elo showing up in the NCM data? I know elo isn't transitive (i.e. if engine A beats engine B by 50 elo and engine B beats engine C by 50 elo, it doesn't mean engine A beats engine C by 100 elo), but this is just one patch, and there were plenty of elo-gaining patches that passed Fishtest. For NCM to show *no progress* for months, in spite of all these patches, doesn't make sense to me.
  
  **Tags:** engines, stockfish
  
  > **Comments:**
  > **username_1**: The patch you mention would appear to simply save some storage ( which could be useful for a lichess server playing thousands of opponents at once, for example ), but would not affect playing strength against a single opponent.  
  > **username_0**: @username_1 isn't fishtest a 1v1 contest against a single opponent too? Also, what about the other patches on that website?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-05-31. Score: 4 | Accepted answer
  
  Fishtest tests proposed changes to Stockfish's code with [SPRT](https://en.wikipedia.org/wiki/Sequential_probability_ratio_test) to determine if they should be accepted. Broadly speaking, there are two types of changes which are tested in this way (non-functional changes do not go through Fishtest): simplifications, and changes which intend to gain strength - elo changes.
  
  The crucial difference here is that simplifications (like the one referenced in the question) are tested for no regression, whereas elo changes are tested for an improvement: SPRT bounds of [0.5, 4.5] are used for elo changes and bounds of [-3, 1] are used for simplifications.
  
  The vast majority of changes to Stockfish recently have been simplifications, and as such they are not expected to increase strength.
  
  For example, the patch in the question is a simplification, and was tested with SPRT bounds [-3.00, 1.00]. The fact that it passed tells us that *there is a 95% chance that the change does not lose 3 elo*. If the test failed, we would know that *there is a 95% chance that the change does not gain 1 elo*.
  
  We can't say anything more precise than this, because SPRT stops immediately as soon as there is enough evidence to accept either hypothesis and doesn't attempt to measure the elo change more precisely. Although an elo difference of +2.18 is quoted at STC, all we know is that the lower error bound on the elo gain is `>= -3.00`.
  
  There is insufficient evidence to say that the change produced an actual elo gain of +2.18, and thus claiming that this simplification gains 2 elo would be misleading.
  
  The small number of elo gaining changes is what results in Stockfish's slow progress at the moment.
  
  > **Comments:**
  > **username_0**: Thanks for answer. Does this mean Stockfish developers classify their patches as "elo gaining" or "simplifications" before submitting them to fishtest?  
  > **username_2**: Stockfish has guidelines for testing methodology which describe the bounds that should be used for different types of changes. Developers submit a test to fishtest with settings that match one of the guidelines, unless they have a good reason to do otherwise. They can then open a pull request, quoting the results of the test.  
  
  ---
  
  