Answers.Body: <p>There's quite a bit to pay attention to when comparing engines: time control, hardware, opening book, etc. In the case of CCRL and Leela, this is the key line:</p>
  <p><a href="https://i.sstatic.net/yk6FV.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/yk6FV.png" alt="enter image description here" /></a></p>
  <p>Leela is a GPU engine - it needs GPU resources to be optimal. This compares to CPU engines, which both Komodo and Stockfish are. It's why hardware was so controversial when Leela first appeared (you can't run Komodo and Stockfish on GPU, and Leela on CPU is much weaker, so how can you have a &quot;fair&quot; match between them?).</p>
  <p>CCRL uses CPU to do their benchmarking, which means Leela is much weaker, and is why Leela is rated so poorly.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-17T11:35:38.307Z
Answers.Text: _"how can you have a "fair" match between them?"_ How'd TCEC figure it out? I think determining comparable compute is entirely possible, and that it's about CCRL being unable to afford hardware. Thanks -- would you know of an alternate leaderboard? (optional question)
Answers.UserDisplayName: OverLordGoldDragon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-17T13:20:44.440Z
Answers.Text: @OverLordGoldDragon There're several different ways you might define "fair". An example is "best commercially available". Another is "equal power consumption". Yet another is "equal cost". To get the exact specifics, you'd have to ask the TCEC admins. As for rating lists, CCRL is the most widely-followed one, but there's also https://www.sp-cc.de/, probably some other lists as well.
Answers.UserDisplayName: Allure
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-17T02:02:16.097Z
Answers.LastActivityDate: 2023-08-17T02:02:16.097Z
Answers.UserDisplayName: Allure
Answers.Body: <p>They test under CPU-only conditions. CPUs are not a good platform for neural network inference (because they are designed to do well on a large variety of workloads, as opposed to more specialized hardware, which can be designed to specifically run neural networks efficiently), and therefore Leela, being an engine heavily relying on neural networks, is much weaker under CPU-only conditions than it is when having access to a good GPU. Basically, the core computational workload of neural networks comes from doing many large matrix-vector multiplications, and graphics hardware is very good at doing those efficiently.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-17T11:33:31.497Z
Answers.Text: Accepted for technical explanation for broader audience.
Answers.UserDisplayName: OverLordGoldDragon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-16T21:42:07.970Z
Answers.LastActivityDate: 2023-08-16T21:42:07.970Z
Answers.UserDisplayName: Polytropos
Body: <p>Lc0 routinely makes it far in the <a href="https://tcec-chess.com/" rel="noreferrer">Top Chess Engine Championship</a> (TCEC). Latest, it defeated Komodo Dragon and nearly tied Stockfish:</p>
  <img src="https://i.sstatic.net/MjOQY.png" width="500">
  <p>Yet, we see on <a href="https://computerchess.org.uk/ccrl/4040/" rel="noreferrer">CCRL 40/15</a>, Lc0 is ranked <strong>#32, 159 ELO behind Komodo</strong>!</p>
  <img src="https://i.sstatic.net/QPNng.png" width="600">
  <p>The gap is greater for 40/2 (#24, 499) but smaller for BLITZ 2+1 (#7, 56).</p>
  <p>I get there's hardware and maybe time control differences, but the gaps seem exaggerated.</p>
  
Tags: <engines><rating><leela>
Title: Why is Leela Chess Zero rated low on CCRL leaderboards?
ThreadText: # Why is Leela Chess Zero rated low on CCRL leaderboards?
  
  (Asked by: username_0 on 2023-08-16)
  
  Lc0 routinely makes it far in the [Top Chess Engine Championship](https://tcec-chess.com/) (TCEC). Latest, it defeated Komodo Dragon and nearly tied Stockfish:
  
  ![](https://i.sstatic.net/MjOQY.png)
  
  Yet, we see on [CCRL 40/15](https://computerchess.org.uk/ccrl/4040/), Lc0 is ranked **#32, 159 ELO behind Komodo**!
  
  ![](https://i.sstatic.net/QPNng.png)
  
  The gap is greater for 40/2 (#24, 499) but smaller for BLITZ 2+1 (#7, 56).
  
  I get there's hardware and maybe time control differences, but the gaps seem exaggerated.
  
  **Tags:** engines, rating, leela
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-08-16. Score: 7 | Accepted answer
  
  They test under CPU-only conditions. CPUs are not a good platform for neural network inference (because they are designed to do well on a large variety of workloads, as opposed to more specialized hardware, which can be designed to specifically run neural networks efficiently), and therefore Leela, being an engine heavily relying on neural networks, is much weaker under CPU-only conditions than it is when having access to a good GPU. Basically, the core computational workload of neural networks comes from doing many large matrix-vector multiplications, and graphics hardware is very good at doing those efficiently.
  
  > **Comments:**
  > **username_0**: Accepted for technical explanation for broader audience.  
  
  ---
  
  ### Answer by username_1 on 2023-08-17. Score: 2
  
  There's quite a bit to pay attention to when comparing engines: time control, hardware, opening book, etc. In the case of CCRL and Leela, this is the key line:
  
  [![enter image description here](https://i.sstatic.net/yk6FV.png)](https://i.sstatic.net/yk6FV.png)
  
  Leela is a GPU engine - it needs GPU resources to be optimal. This compares to CPU engines, which both Komodo and Stockfish are. It's why hardware was so controversial when Leela first appeared (you can't run Komodo and Stockfish on GPU, and Leela on CPU is much weaker, so how can you have a "fair" match between them?).
  
  CCRL uses CPU to do their benchmarking, which means Leela is much weaker, and is why Leela is rated so poorly.
  
  > **Comments:**
  > **username_0**: _"how can you have a "fair" match between them?"_ How'd TCEC figure it out? I think determining comparable compute is entirely possible, and that it's about CCRL being unable to afford hardware. Thanks -- would you know of an alternate leaderboard? (optional question)  
  > **username_1**: @username_0 There're several different ways you might define "fair". An example is "best commercially available". Another is "equal power consumption". Yet another is "equal cost". To get the exact specifics, you'd have to ask the TCEC admins. As for rating lists, CCRL is the most widely-followed one, but there's also https://www.sp-cc.de/, probably some other lists as well.  
  
  ---
  
  