Answers.Body: <p>The point of fishtesting is to test if a newer version of Stockfish has become <em>stronger</em>. The definition of stronger is it beating its previous version.</p>
  
  <p>I'm not sure how testing each new version against an engine like Leela would help. Your idea might be only accepting a new version of Stockfish if it performs better against Leela than the previous version of Stockfish did. So now each version of Stockfish is guaranteed to play better against Leela, but not guaranteed to be a better chess player overall. The reason is that Stockfish could rely on certain characteristics in Leela's play to win more games.</p>
  
  <p>EDIT - I thought of an example that might help here. Suppose you were rated 1500 a few years ago, and now you're 1800. If you played your past self in a match it's clear you would win. But for the sake of argument it's possible your 1500-self would perform better against Kasparov than your current 1800 self. Maybe the 1500-self was less conservative and took way more risks. This manages to beat Kasparov in a rare game, while your current 1800 self would lose every time (yet the games would be slightly closer on average).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-07T13:31:29.037Z
Answers.Text: And in the end, "Slightly closer" would be hard to quantify. Self-play is best.
Answers.UserDisplayName: Brandon_J
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-07T15:13:09.897Z
Answers.Text: Seems like semantics, but what makes you say "stronger" is defined as beating the previous version? One could equally argue that "stronger" is the guy that beat Kasparov, wins the tournament, and gets all the glory, not the one that wins the head-to-head.
Answers.UserDisplayName: Allure
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-07T18:49:11.287Z
Answers.Text: @Allure I think head-to-head is just the most objective measure of "strength". If A beats B, A played better than B, ergo A is stronger than B. But if A does slightly less worse against C than B did against C, does that necessarily mean A > B? What if B performed better against D, E, F, G... than A did?
Answers.UserDisplayName: Inertial Ignorance
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-07T18:51:52.267Z
Answers.Text: But you have a point. The Candidates system for the World Championship is a big tournament, but there used to be individual matches between players. For a big tournament A could lose to B but still win the tournament and be declared "strongest". However, this is because A beat a variety of different players, not just one player C. So maybe engine testing could be used where the newer version plays a variety of different engines, but it's faster and simpler to just test against the older version.
Answers.UserDisplayName: Inertial Ignorance
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-07T04:27:42.847Z
Answers.LastActivityDate: 2019-06-07T04:58:01.823Z
Answers.LastEditDate: 2019-06-07T04:58:01.823Z
Answers.UserDisplayName: Inertial Ignorance
Answers.Body: <p>I'll agree in the main with @inertialignorance but I'd like to clarify the position a bit.</p>
  
  <p>When human plays human, the result of one game is relatively meaningless, given the extreme variability of human play. (I'll use Kasparov v Deep Blue as an example of that variability -- on a good day <em>I</em> probably could have beaten Kasparov in the game he blundered away against DB.) So let's just ignore humanity in this and concentrate on the machine.</p>
  
  <p>Machines play at a far more consistent level of skill than humans. Therefore a single game means more, and a series of games means a lot. So it makes sense to test the proposed Stockfish improvement against a machine. But why Stockfish?</p>
  
  <p>Simply because there's no way to quantify "does better against Leela" in any meaningful way. It could do better, and still lose. But what's the standard for better, in that case? How do you prove it played better? I can't see a viable standard.</p>
  
  <p>It's far simpler to prove Stockfish prime is an improvement over Stockfish original by simply playing a set of games and counting the results.</p>
  
  <p>Edward Deming maintained that it was of prime importance to chose the correct metric, because you only know you improve because of what you measure. So ask the question: What is the purpose of the process? Why are you proposing a patch for Stockfish?</p>
  
  <p>Is it really the ultimate goal of the patch to beat Leela? Or is it rather to make Stockfish play better chess? I'd say it's the latter. Beating Leela will come on its own if only Stockfish continues to improve enough.</p>
  
  <p>Yes, an incremental improvement path for Stockfish could come to a plateau. Even so, the only verifiable path off that plateau will lie in making changes to it that make it better. If an approach "hits a wall," then a continual search for improvements will dictate another path. If a proposed alternate path can't beat the current path, why choose it?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-11T22:19:07.997Z
Answers.Text: *How do you prove it played better?* it should be pretty straightforward, no? Just play the previous version 20k times against Leela, and the new version 20k times as well, then compare the results.
Answers.UserDisplayName: Allure
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-12T00:30:21.797Z
Answers.Text: @Allure But hypothetically, what if Stockfish Prime beat old Stockfish, but old Stockfish did better against Leela (and as Arlen mentioned, how do you even compare which Stockfish "did better" against Leela). Would you not keep Stockfish prime then? This verification process seems flawed.
Answers.UserDisplayName: Inertial Ignorance
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-12T00:36:55.263Z
Answers.Text: @InertialIgnorance Equally, what if Stockfish Prime did better against Leela, but failed to beat old Stockfish, would you not keep Stockfish Prime then? It comes down to how one defines "stronger", and it's not clear to me that "stronger" is winning the head-to-head.
Answers.UserDisplayName: Allure
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-12T02:55:54.963Z
Answers.Text: @Allure I'll admit there's still a reason to be conflicted there, but in that case it feels more clear to prefer Stockfish Prime. When you want to compare which two things are better (be them politicians, athletes, etc), you most often get them to compete against each other in some fashion. Not to see who did better against an arbitrarily selected peer.
Answers.UserDisplayName: Inertial Ignorance
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-12T04:01:22.623Z
Answers.Text: @InertialIgnorance still, if Karjakin had won the 2016 World Championships, it's hard for me to imagine that most people will consider him the strongest player in the world. Yes, he'd have beaten Carlsen in the head-to-head match, but Carlsen's tournament victories would indicate the reverse. Similarly when Kasparov wasn't world champion, most people still considered him the strongest.
Answers.UserDisplayName: Allure
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-12T04:01:59.287Z
Answers.Text: Don't think it's fair to say Leela is an "arbitrarily selected peer" either - it's the current TCEC champion after all, and if Leela is ever dethroned it would (should?) be simple to move to the next champion as the sparring partner.
Answers.UserDisplayName: Allure
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-13T23:37:46.397Z
Answers.Text: @Allure But the point of fishtesting isn't to ensure the newer version of Stockfish is the strongest engine in the world. It just tries to verify that it's most likely stronger than the previous version of Stockfish.
Answers.UserDisplayName: Inertial Ignorance
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-19T20:13:31.353Z
Answers.Text: It's possible for stockfish prime to play better chess and still have the same score against Leela. It could show better positional understanding, for example, and yet not be enough better to change the outcomes. Yet it would defeat old stockfish in a match because of that improvement. And likewise it's possible the improvement shows up under circumstances that don't occur against Leela; chess positions aren't randomly distributed; they grow out of deliberate choices.
Answers.UserDisplayName: Arlen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-11T20:23:26.140Z
Answers.LastActivityDate: 2019-06-11T20:23:26.140Z
Answers.UserDisplayName: Arlen
Answers.Body: <p>There appears to be a hardware reason for not making this change. </p>
  
  <p>The main problem with using Leela as a sparring opponent is that Leela runs best on GPU. It's possible to run Leela on CPU, but Leela's performance suffers a lot. The OP suggests giving Leela time odds to compensate, but time odds does not work very well: Leela's performance weakens so much that the time odds required are oppressively long. </p>
  
  <p>To get an idea of how much weaker Leela is on CPU, we can look at Leela in season 12 of TCEC, when it didn't have GPU support and ran on CPU. <a href="https://cd.tcecbeta.club/archive.html?season=12&amp;div=4&amp;game=17" rel="nofollow noreferrer">Here's an example game played by Leela then</a>. If one looks at the speeds it was achieving, it's about 1-3kn/s, or 1000-3000 positions per second. Comparatively, in the latest season 15 when Leela was running on powerful GPUs, it would achieve about 50kn/s (<a href="https://cd.tcecbeta.club/archive.html?season=15&amp;div=sf&amp;game=13" rel="nofollow noreferrer">example game</a>). Therefore to be able to test Stockfish against Leela on equal terms, one needs to give Leela about 25x time odds. If Stockfish has one minute, Leela needs 25 minutes. </p>
  
  <p>As of time of writing, Fishtesting tests at two time controls: 10s + 0.1s/move, and 60s + 0.6s/move (patches that pass the first, short time control test get promoted to the longer one and tested again. Patches that pass the second then become the "new" version). At 25x time odds, Leela needs 250s + 2.5s/move in the first time control and 1500s + 15s/move in the second time control. The slowdown is tremendous; we would effectively have Leela playing at rapid time controls and not bullet. The number of games that can be completed in unit time would also drop by about 25x. Fishtesting regularly needs tens of thousands of games to test each patch; taking 25x as long to finish each test sounds unacceptably slow.</p>
  
  <p>To add fuel to the fire, as far as I know, Leela had a smaller network in season 12 - the latest nets' performance could be even slower on CPU now.</p>
  
  <p>It's possible that someday, if Fishtesting gets the GPU resources to test with Leela, it might switch over; however we are not at that point yet.</p>
  
  <p><strong>Edit</strong>: Lc0-CPU is currently playing at TCEC. It gets about 5knps. According to some people in the chat though, Lc0-CPU is modified to play on CPU; unmodified, it is about 80x slower than it is on GPU. Therefore there's a real hardware cost to using Lc0-CPU as a testing opponent.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-16T11:01:34.863Z
Answers.Text: I doubt that this will happen. Almost all engine authors are mainly testing their engine against itself and not against other (possibly stronger) engines. It might be valuable to in addition do cross-checks by testing against other engines, but this should not be the main part of testing, because it is less efficient.
Answers.UserDisplayName: Fabian Fichter
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-16T12:12:18.233Z
Answers.Text: @FabianFichter why is it less efficient to test against other engines?
Answers.UserDisplayName: Allure
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-16T13:45:45.057Z
Answers.Text: As I described in my answer (https://chess.stackexchange.com/a/24714/15415) it is less efficient because you need more games (i.e., more hardware or time) to reach the same statistical precision for your Elo measurement.
Answers.UserDisplayName: Fabian Fichter
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-14T02:06:54.077Z
Answers.LastActivityDate: 2019-07-19T03:29:45.113Z
Answers.LastEditDate: 2019-07-19T03:29:45.113Z
Answers.UserDisplayName: Allure
Answers.Body: <p>(I'm adding another answer because it's not actually mine)</p>
  
  <p>A Stockfish developer by the name of Михаил Чалый wrote <a href="https://groups.google.com/d/msg/fishcooking/mk4ecwtNW-I/EznU86GBAQAJ" rel="nofollow noreferrer">this</a> on the Fishcooking Google Groups (basically a forum for Stockfish developers) when this same idea was suggested there:</p>
  
  <blockquote>
    <p>1) We don't have hardware suitable for lc0 on most of the machines (80+% of machines are noob machines* w/o GPU);</p>
    
    <p>2) testing against external engine will double error bars thus increase amount of games needing for test to be reliable by factor of 4;</p>
    
    <p>3) you will need to write new logic since you a) can't use SPRT, b) you need to normalize leela/sf speed to some value (as we do with TC in normal tests) so you will need assymmetric time controls;</p>
    
    <p>4) there are 0 proof that patches that work in selfplay don't work vs leela. Actually there are opposite proofs like fastgm 16 cores list where sf 11 is ahead of sf10 by 50 elo which is exactly what we measure if you match them against each other.</p>
    
    <p>So tldr - it's hard or almost impossible to do and gains of this are not really existing anyway.</p>
  </blockquote>
  
  <p>*As far as I'm aware, "noob machines" are machines owned by another person in the community named "noob", who has lots of CPU hardware available but not GPU hardware.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-04T03:26:10.243Z
Answers.LastActivityDate: 2020-06-04T03:48:28.520Z
Answers.LastEditDate: 2020-06-04T03:48:28.520Z
Answers.UserDisplayName: Allure
Answers.Body: <p>Inertial is correct. I should also add there was no strong open source engine before LC0. Both Komodo and Houdini had licensing restriction.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-10T00:58:34.597Z
Answers.Text: "no strong open source engine" I suppose you mean _besides_ stockfish?
Answers.UserDisplayName: Brandon_J
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-11T02:19:10.943Z
Answers.Text: @Brandon_J yes that was what I meant
Answers.UserDisplayName: SmallChess
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-07T13:48:00.263Z
Answers.LastActivityDate: 2019-06-07T13:48:00.263Z
Answers.UserDisplayName: SmallChess
Answers.Body: <p>First of all, what does not become clear from other answers is that you of course can in principle use any opponent (with a consistent playing strength) to compare the playing strength of two versions of a program.</p>
  
  <p>However, there are several good reasons to test the versions directly against each other, regardless of whether you are developing Stockfish or any other engine:</p>
  
  <ul>
  <li>The playing strength comparison of chess playing entities usually does not fulfill transitivity, i.e., if A > B and B > C, A &lt; C is still possible. So as long as a comparison against a large pool of opponents (in order to average out the transitivity issue) is not feasible, a direct comparison should be the most reliable measurement, although you can of course have the same issue for three versions of a single program.</li>
  <li>By doing a direct comparison, you reduce the required number of games in testing, because you only need to run one match instead of two, and at the same time you get smaller error bars for the playing strength difference.</li>
  <li>Testing against an opponent of (almost) equal strength maximizes the statistical sensitivity of a single game result, so you again save hardware costs. If the difference in playing strength is very high, the information/entropy of a single game is very low.</li>
  </ul>
  
  <p>Furthermore, in the case of fishtest/Stockfish there are several concrete reasons why choosing a direct comparison was the most principled choice:</p>
  
  <ul>
  <li>At the time fishtest was initially developed, Leela was not available.</li>
  <li>Using commercial engines such as Komodo and Houdini was/is not an option because of licensing issues.</li>
  <li>Using other programs can introduce security risks as well as additional software dependencies or hardware requirements you want to avoid in a distributed computing environment.</li>
  <li>Previous to Leela, there basically was no open source engine that was strong enough in order to get statistically significant results when matching against Stockfish.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-17T03:56:06.540Z
Answers.Text: Are you sure testing against a 3rd engine increases the number of games required? It seems like it only does so the first time. For example let's say the third engine is E, and the versions of Stockfish being tried are A, B and C. We play A and B 20k times against E and get a score. Whichever does better is kept and the weaker one rejected. Say A is kept. When it comes to testing version C, we already have the A-E results, so we only need to get C-E results, and the same applies to all future versions.
Answers.UserDisplayName: Allure
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-17T09:08:53.197Z
Answers.Text: Even if you always test against the same version of your reference engine (which makes it very prone to overfitting), the statistical uncertainty on the resulting Elo difference will still be bigger, because by combining the results from two runs (e.g., EloC - EloB) you also need to [combine the statistical uncertainties](https://en.wikipedia.org/wiki/Propagation_of_uncertainty), which usually increases the uncertainty by a factor of sqrt(2). In order to compensate for that, you would need to run more games.
Answers.UserDisplayName: Fabian Fichter
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-14T09:26:26.477Z
Answers.LastActivityDate: 2019-06-14T09:31:43.853Z
Answers.LastEditDate: 2019-06-14T09:31:43.853Z
Answers.UserDisplayName: Fabian Fichter
Body: <p>As I understand the so-called <a href="http://tests.stockfishchess.org/tests" rel="noreferrer">Fishtesting</a>, people write patches, then try the patches against the current version of Stockfish. If the new version performs better, it's promoted to be the main version; otherwise it's rejected.</p>
  
  <p>Question: why test against the current version of Stockfish? It's worked in the past, and current Stockfish is a lot stronger than two-year old Stockfish. However, we can also see limitations to this approach. Although Stockfish is now capable of utterly destroying other traditional engines, it also recently lost the TCEC S15 superfinal against Leela. Therefore, why not test each new version against Leela? </p>
  
  <p>The obvious answer is that Fishtesting's resources come from CPUs, and Leela is awful on CPU. However that shouldn't be fatal: one could just give Leela more time, e.g. with Stockfish running on 10s per game, give Leela 100s per game or whatever time odds are necessary to get the Leela ratio to the ratio in the superfinal. This will certainly slow down Fishtesting, but if it leads to an engine that can beat Leela in a match, that might still be worth it.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-18T18:37:49.383Z
Comments.Text: Most likely because Stockfish is easily available and modifyable to Stockfish developers.
Comments.UserDisplayName: Thorbjørn Ravn Andersen
Tags: <engines><stockfish>
Title: Why does Fishtesting test Stockfish against itself, not other engines?
ThreadText: # Why does Fishtesting test Stockfish against itself, not other engines?
  
  (Asked by: username_0 on 2019-06-07)
  
  As I understand the so-called [Fishtesting](http://tests.stockfishchess.org/tests), people write patches, then try the patches against the current version of Stockfish. If the new version performs better, it's promoted to be the main version; otherwise it's rejected.
  
  Question: why test against the current version of Stockfish? It's worked in the past, and current Stockfish is a lot stronger than two-year old Stockfish. However, we can also see limitations to this approach. Although Stockfish is now capable of utterly destroying other traditional engines, it also recently lost the TCEC S15 superfinal against Leela. Therefore, why not test each new version against Leela?
  
  The obvious answer is that Fishtesting's resources come from CPUs, and Leela is awful on CPU. However that shouldn't be fatal: one could just give Leela more time, e.g. with Stockfish running on 10s per game, give Leela 100s per game or whatever time odds are necessary to get the Leela ratio to the ratio in the superfinal. This will certainly slow down Fishtesting, but if it leads to an engine that can beat Leela in a match, that might still be worth it.
  
  **Tags:** engines, stockfish
  
  > **Comments:**
  > **username_1**: Most likely because Stockfish is easily available and modifyable to Stockfish developers.  
  
  ---
  
  ## 6 answers
  
  ### Answer by username_5 on 2019-06-14. Score: 6 | Accepted answer
  
  First of all, what does not become clear from other answers is that you of course can in principle use any opponent (with a consistent playing strength) to compare the playing strength of two versions of a program.
  
  However, there are several good reasons to test the versions directly against each other, regardless of whether you are developing Stockfish or any other engine:
  
  * The playing strength comparison of chess playing entities usually does not fulfill transitivity, i.e., if A > B and B > C, A < C is still possible. So as long as a comparison against a large pool of opponents (in order to average out the transitivity issue) is not feasible, a direct comparison should be the most reliable measurement, although you can of course have the same issue for three versions of a single program.
  * By doing a direct comparison, you reduce the required number of games in testing, because you only need to run one match instead of two, and at the same time you get smaller error bars for the playing strength difference.
  * Testing against an opponent of (almost) equal strength maximizes the statistical sensitivity of a single game result, so you again save hardware costs. If the difference in playing strength is very high, the information/entropy of a single game is very low.
  
  Furthermore, in the case of fishtest/Stockfish there are several concrete reasons why choosing a direct comparison was the most principled choice:
  
  * At the time fishtest was initially developed, Leela was not available.
  * Using commercial engines such as Komodo and Houdini was/is not an option because of licensing issues.
  * Using other programs can introduce security risks as well as additional software dependencies or hardware requirements you want to avoid in a distributed computing environment.
  * Previous to Leela, there basically was no open source engine that was strong enough in order to get statistically significant results when matching against Stockfish.
  
  > **Comments:**
  > **username_0**: Are you sure testing against a 3rd engine increases the number of games required? It seems like it only does so the first time. For example let's say the third engine is E, and the versions of Stockfish being tried are A, B and C. We play A and B 20k times against E and get a score. Whichever does better is kept and the weaker one rejected. Say A is kept. When it comes to testing version C, we already have the A-E results, so we only need to get C-E results, and the same applies to all future versions.  
  > **username_5**: Even if you always test against the same version of your reference engine (which makes it very prone to overfitting), the statistical uncertainty on the resulting Elo difference will still be bigger, because by combining the results from two runs (e.g., EloC - EloB) you also need to [combine the statistical uncertainties](https://en.wikipedia.org/wiki/Propagation_of_uncertainty), which usually increases the uncertainty by a factor of sqrt(2). In order to compensate for that, you would need to run more games.  
  
  ---
  
  ### Answer by username_2 on 2019-06-07. Score: 7
  
  The point of fishtesting is to test if a newer version of Stockfish has become *stronger*. The definition of stronger is it beating its previous version.
  
  I'm not sure how testing each new version against an engine like Leela would help. Your idea might be only accepting a new version of Stockfish if it performs better against Leela than the previous version of Stockfish did. So now each version of Stockfish is guaranteed to play better against Leela, but not guaranteed to be a better chess player overall. The reason is that Stockfish could rely on certain characteristics in Leela's play to win more games.
  
  EDIT - I thought of an example that might help here. Suppose you were rated 1500 a few years ago, and now you're 1800. If you played your past self in a match it's clear you would win. But for the sake of argument it's possible your 1500-self would perform better against Kasparov than your current 1800 self. Maybe the 1500-self was less conservative and took way more risks. This manages to beat Kasparov in a rare game, while your current 1800 self would lose every time (yet the games would be slightly closer on average).
  
  > **Comments:**
  > **username_3**: And in the end, "Slightly closer" would be hard to quantify. Self-play is best.  
  > **username_0**: Seems like semantics, but what makes you say "stronger" is defined as beating the previous version? One could equally argue that "stronger" is the guy that beat Kasparov, wins the tournament, and gets all the glory, not the one that wins the head-to-head.  
  > **username_2**: @username_0 I think head-to-head is just the most objective measure of "strength". If A beats B, A played better than B, ergo A is stronger than B. But if A does slightly less worse against C than B did against C, does that necessarily mean A > B? What if B performed better against D, E, F, G... than A did?  
  > **username_2**: But you have a point. The Candidates system for the World Championship is a big tournament, but there used to be individual matches between players. For a big tournament A could lose to B but still win the tournament and be declared "strongest". However, this is because A beat a variety of different players, not just one player C. So maybe engine testing could be used where the newer version plays a variety of different engines, but it's faster and simpler to just test against the older version.  
  
  ---
  
  ### Answer by username_4 on 2019-06-11. Score: 3
  
  I'll agree in the main with @username_2 but I'd like to clarify the position a bit.
  
  When human plays human, the result of one game is relatively meaningless, given the extreme variability of human play. (I'll use Kasparov v Deep Blue as an example of that variability -- on a good day *I* probably could have beaten Kasparov in the game he blundered away against DB.) So let's just ignore humanity in this and concentrate on the machine.
  
  Machines play at a far more consistent level of skill than humans. Therefore a single game means more, and a series of games means a lot. So it makes sense to test the proposed Stockfish improvement against a machine. But why Stockfish?
  
  Simply because there's no way to quantify "does better against Leela" in any meaningful way. It could do better, and still lose. But what's the standard for better, in that case? How do you prove it played better? I can't see a viable standard.
  
  It's far simpler to prove Stockfish prime is an improvement over Stockfish original by simply playing a set of games and counting the results.
  
  Edward Deming maintained that it was of prime importance to chose the correct metric, because you only know you improve because of what you measure. So ask the question: What is the purpose of the process? Why are you proposing a patch for Stockfish?
  
  Is it really the ultimate goal of the patch to beat Leela? Or is it rather to make Stockfish play better chess? I'd say it's the latter. Beating Leela will come on its own if only Stockfish continues to improve enough.
  
  Yes, an incremental improvement path for Stockfish could come to a plateau. Even so, the only verifiable path off that plateau will lie in making changes to it that make it better. If an approach "hits a wall," then a continual search for improvements will dictate another path. If a proposed alternate path can't beat the current path, why choose it?
  
  > **Comments:**
  > **username_0**: *How do you prove it played better?* it should be pretty straightforward, no? Just play the previous version 20k times against Leela, and the new version 20k times as well, then compare the results.  
  > **username_2**: @username_0 But hypothetically, what if Stockfish Prime beat old Stockfish, but old Stockfish did better against Leela (and as username_4 mentioned, how do you even compare which Stockfish "did better" against Leela). Would you not keep Stockfish prime then? This verification process seems flawed.  
  > **username_0**: @username_2 Equally, what if Stockfish Prime did better against Leela, but failed to beat old Stockfish, would you not keep Stockfish Prime then? It comes down to how one defines "stronger", and it's not clear to me that "stronger" is winning the head-to-head.  
  > **username_2**: @username_0 I'll admit there's still a reason to be conflicted there, but in that case it feels more clear to prefer Stockfish Prime. When you want to compare which two things are better (be them politicians, athletes, etc), you most often get them to compete against each other in some fashion. Not to see who did better against an arbitrarily selected peer.  
  > **username_0**: @username_2 still, if Karjakin had won the 2016 World Championships, it's hard for me to imagine that most people will consider him the strongest player in the world. Yes, he'd have beaten Carlsen in the head-to-head match, but Carlsen's tournament victories would indicate the reverse. Similarly when Kasparov wasn't world champion, most people still considered him the strongest.  
  > **username_0**: Don't think it's fair to say Leela is an "arbitrarily selected peer" either - it's the current TCEC champion after all, and if Leela is ever dethroned it would (should?) be simple to move to the next champion as the sparring partner.  
  > **username_2**: @username_0 But the point of fishtesting isn't to ensure the newer version of Stockfish is the strongest engine in the world. It just tries to verify that it's most likely stronger than the previous version of Stockfish.  
  > **username_4**: It's possible for stockfish prime to play better chess and still have the same score against Leela. It could show better positional understanding, for example, and yet not be enough better to change the outcomes. Yet it would defeat old stockfish in a match because of that improvement. And likewise it's possible the improvement shows up under circumstances that don't occur against Leela; chess positions aren't randomly distributed; they grow out of deliberate choices.  
  
  ---
  
  ### Answer by username_0 on 2019-06-14. Score: 3
  
  There appears to be a hardware reason for not making this change.
  
  The main problem with using Leela as a sparring opponent is that Leela runs best on GPU. It's possible to run Leela on CPU, but Leela's performance suffers a lot. The OP suggests giving Leela time odds to compensate, but time odds does not work very well: Leela's performance weakens so much that the time odds required are oppressively long.
  
  To get an idea of how much weaker Leela is on CPU, we can look at Leela in season 12 of TCEC, when it didn't have GPU support and ran on CPU. [Here's an example game played by Leela then](https://cd.tcecbeta.club/archive.html?season=12&div=4&game=17). If one looks at the speeds it was achieving, it's about 1-3kn/s, or 1000-3000 positions per second. Comparatively, in the latest season 15 when Leela was running on powerful GPUs, it would achieve about 50kn/s ([example game](https://cd.tcecbeta.club/archive.html?season=15&div=sf&game=13)). Therefore to be able to test Stockfish against Leela on equal terms, one needs to give Leela about 25x time odds. If Stockfish has one minute, Leela needs 25 minutes.
  
  As of time of writing, Fishtesting tests at two time controls: 10s + 0.1s/move, and 60s + 0.6s/move (patches that pass the first, short time control test get promoted to the longer one and tested again. Patches that pass the second then become the "new" version). At 25x time odds, Leela needs 250s + 2.5s/move in the first time control and 1500s + 15s/move in the second time control. The slowdown is tremendous; we would effectively have Leela playing at rapid time controls and not bullet. The number of games that can be completed in unit time would also drop by about 25x. Fishtesting regularly needs tens of thousands of games to test each patch; taking 25x as long to finish each test sounds unacceptably slow.
  
  To add fuel to the fire, as far as I know, Leela had a smaller network in season 12 - the latest nets' performance could be even slower on CPU now.
  
  It's possible that someday, if Fishtesting gets the GPU resources to test with Leela, it might switch over; however we are not at that point yet.
  
  **Edit**: Lc0-CPU is currently playing at TCEC. It gets about 5knps. According to some people in the chat though, Lc0-CPU is modified to play on CPU; unmodified, it is about 80x slower than it is on GPU. Therefore there's a real hardware cost to using Lc0-CPU as a testing opponent.
  
  > **Comments:**
  > **username_5**: I doubt that this will happen. Almost all engine authors are mainly testing their engine against itself and not against other (possibly stronger) engines. It might be valuable to in addition do cross-checks by testing against other engines, but this should not be the main part of testing, because it is less efficient.  
  > **username_0**: @username_5 why is it less efficient to test against other engines?  
  > **username_5**: As I described in my answer (https://chess.stackexchange.com/a/24714/15415) it is less efficient because you need more games (i.e., more hardware or time) to reach the same statistical precision for your Elo measurement.  
  
  ---
  
  ### Answer by username_0 on 2020-06-04. Score: 2
  
  (I'm adding another answer because it's not actually mine)
  
  A Stockfish developer by the name of Михаил Чалый wrote [this](https://groups.google.com/d/msg/fishcooking/mk4ecwtNW-I/EznU86GBAQAJ) on the Fishcooking Google Groups (basically a forum for Stockfish developers) when this same idea was suggested there:
  
  > 1) We don't have hardware suitable for lc0 on most of the machines (80+% of machines are noob machines\* w/o GPU);
  >
  > 2) testing against external engine will double error bars thus increase amount of games needing for test to be reliable by factor of 4;
  >
  > 3) you will need to write new logic since you a) can't use SPRT, b) you need to normalize leela/sf speed to some value (as we do with TC in normal tests) so you will need assymmetric time controls;
  >
  > 4) there are 0 proof that patches that work in selfplay don't work vs leela. Actually there are opposite proofs like fastgm 16 cores list where sf 11 is ahead of sf10 by 50 elo which is exactly what we measure if you match them against each other.
  >
  > So tldr - it's hard or almost impossible to do and gains of this are not really existing anyway.
  
  \*As far as I'm aware, "noob machines" are machines owned by another person in the community named "noob", who has lots of CPU hardware available but not GPU hardware.
  
  
  ---
  
  ### Answer by username_6 on 2019-06-07. Score: 0
  
  username_2 is correct. I should also add there was no strong open source engine before LC0. Both Komodo and Houdini had licensing restriction.
  
  > **Comments:**
  > **username_3**: "no strong open source engine" I suppose you mean _besides_ stockfish?  
  > **username_6**: @username_3 yes that was what I meant  
  
  ---
  
  