Answers.Body: <p>Stockfish 16 UCI_Elo was <a href="https://github.com/official-stockfish/Stockfish/commit/a08b8d4" rel="nofollow noreferrer">calibrated</a> with the <strong>CCRL rating pool</strong>. See <a href="https://chess.stackexchange.com/questions/25328/are-engine-ratings-directly-comparable-to-human-level-play">Are engine ratings directly comparable to human-level play?</a> From the <a href="https://disservin.github.io/stockfish-docs/stockfish-wiki/Stockfish-FAQ.html#the-elo-rating-of-stockfish" rel="nofollow noreferrer">Stockfish FAQ</a>: &quot;Finally, rating Stockfish on a human scale (e.g. FIDE Elo) has become an almost impossible task, as strength differences between engines and humans are now so large, that this difference can hardly be measured.&quot;</p>
  <pre><code>   # PLAYER             :  RATING  ERROR  POINTS  PLAYED   (%)
     1 master-skill-19    :  3191.1   40.4   940.0    1707    55
     2 master-skill-18    :  3170.3   39.3  1343.0    2519    53
     3 master-skill-17    :  3141.3   37.8  2282.0    4422    52
     4 master-skill-16    :  3111.2   37.1  2773.0    5423    51
     5 master-skill-15    :  3069.5   37.2  2728.5    5386    51
     6 master-skill-14    :  3024.8   36.1  2702.0    5339    51
     7 master-skill-13    :  2972.9   35.4  2645.5    5263    50
     8 master-skill-12    :  2923.1   35.0  2653.5    5165    51
     9 master-skill-11    :  2855.5   33.6  2524.0    5081    50
    10 master-skill-10    :  2788.3   32.0  2724.5    5511    49
    11 stash-bot-v25.0    :  2744.0   31.5  1952.5    3840    51
    12 master-skill-9     :  2702.8   30.5  2670.0    5018    53
    13 master-skill-8     :  2596.2   28.5  2669.5    4975    54
    14 stash-bot-v21.0    :  2561.2   30.0  1338.0    3366    40
    15 master-skill-7     :  2499.5   28.5  1934.0    4178    46
    16 stash-bot-v20.0    :  2452.6   27.7  1606.5    3378    48
    17 stash-bot-v19.0    :  2425.3   26.7  1787.0    3365    53
    18 master-skill-6     :  2363.2   26.4  2510.5    4379    57
    19 stash-bot-v17.0    :  2280.7   25.4  2209.0    4378    50
    20 master-skill-5     :  2203.7   25.3  2859.5    5422    53
    21 stash-bot-v15.3    :  2200.0   25.4  1757.0    4383    40
    22 stash-bot-v14      :  2145.9   25.5  2890.0    5167    56
    23 stash-bot-v13      :  2042.7   25.8  2263.5    4363    52
    24 stash-bot-v12      :  1963.4   25.8  1769.5    4210    42
    25 master-skill-4     :  1922.9   25.9  2690.0    5399    50
    26 stash-bot-v11      :  1873.0   26.3  2203.5    4335    51
    27 stash-bot-v10      :  1783.8   27.8  2568.5    4301    60
    28 master-skill-3     :  1742.3   27.8  1909.5    4439    43
    29 master-skill-2     :  1608.4   29.4  2064.5    4389    47
    30 stash-bot-v9       :  1582.6   30.2  2130.0    4230    50
    31 master-skill-1     :  1467.6   31.3  2015.5    4244    47
    32 stash-bot-v8       :  1452.8   31.5  1953.5    3780    52
    33 master-skill-0     :  1320.1   32.9   651.5    2083    31
  </code></pre>
  <hr />
  <p>There was some discussion on improving Stockfish levels to play more humanlike <a href="https://github.com/official-stockfish/Stockfish/issues/3635" rel="nofollow noreferrer">https://github.com/official-stockfish/Stockfish/issues/3635</a></p>
  <p>dav1312 provides this ELO rating list, but it's not listed how it was derived:</p>
  <pre><code>0    1347
  1    1490
  2    1597
  3    1694
  4    1785
  5    1871
  6    1954
  7    2035
  8    2113
  9    2189
  10    2264
  11    2337
  12    2409
  13    2480
  14    2550
  15    2619
  16    2686
  17    2754
  18    2820
  19    2886
  </code></pre>
  <p>I think lichess could very well keep track of rating based on all the people that play it like for other bots: <a href="https://lichess.org/player/bots" rel="nofollow noreferrer">https://lichess.org/player/bots</a>. I don't think they do, though.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-27T06:49:25.503Z
Answers.Text: In the absence of the ability for me to mark sopel's comment as the correct answer because they didn't submit it as an answer, I'll award it to this one as it's very close to the same essence
Answers.UserDisplayName: day
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-26T21:56:23.977Z
Answers.LastActivityDate: 2024-02-19T17:52:12.060Z
Answers.LastEditDate: 2024-02-19T17:52:12.060Z
Answers.UserDisplayName: qwr
Answers.Body: <p>I am rated 1410 USCF.  I played all 8 Stockfish levels this past weekend, once with each color.  My record was 11-5.
  I was 2-0 vs Stockfish 1-5, 1-1 vs Stockfish 6, and 0-2 vs Stockfish 7 and 8.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-05T03:53:37.500Z
Answers.Text: The most concrete answer, based on experience. Only that lacks the reflection time, the hardware were it runs (MHz and MB of RAM) and if pondering was active, to be more complete.
Answers.UserDisplayName: djnavas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-05T13:29:15.190Z
Answers.Text: Sopel gave the best answer some months back [here](https://chess.stackexchange.com/questions/29860/is-there-a-list-of-approximate-elo-ratings-for-each-stockfish-level/33613#comment47749_29860) but they did it as a comment that I can't mark as the accepted answer. The lichess stockfish bots only have 8 discrete difficulty options and do not surface where on the actual stockfish difficulty setting scale from 1-20 they correspond to, rendering a winrate ELO analysis of them useless for the purposes of determining strength of a direct interface to the engine.
Answers.UserDisplayName: day
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-04T12:02:29.250Z
Answers.LastActivityDate: 2021-01-04T12:02:29.250Z
Answers.UserDisplayName: Eric
Answers.Body: <p>They have a development pipeline doing testing based on one of the early versions.
  I saw a graph of such test in some heavy loading page from Next Chess Move website:</p>
  
  <ul>
  <li><a href="https://nextchessmove.com/dev-builds" rel="nofollow noreferrer">Stockfish Development Builds</a></li>
  </ul>
  
  <p>The development page gathering the test linked from the Fishtest wiki:</p>
  
  <ul>
  <li><a href="https://tests.stockfishchess.org/tests" rel="nofollow noreferrer">Stockfish Testing Queue</a></li>
  </ul>
  
  <p>And for completion:</p>
  
  <ul>
  <li><a href="https://github.com/glinscott/fishtest/wiki" rel="nofollow noreferrer">Fishtest GitHub Repository Wiki</a></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-08T02:51:42.090Z
Answers.Text: These resources seem to be for evaluating optimisations to the engine in development, I don't see any tests evaluating the engine set to levels 1-19, not sure that fishtest would be receptive to having levels 1-19 put in the system given their stated goals
Answers.UserDisplayName: day
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-09T02:27:29.840Z
Answers.Text: I have not looked into the details, but if I were testing for possible regression or improvements, i would use the whole dynamic range available (both sides), I would expect that somewhere is that mass of data there should be a way to get some approximate ELO relationship to the parameter volume explored.  There was no answer.  I hope there will be a better answer later on.  I am also curious.  Do you know the stated goals?
Answers.UserDisplayName: dbdb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-10T06:36:56.523Z
Answers.Text: The results in the first two links don't seem to have any examples of stockfish set to any level besides the highest one. The stated goals of fishtest by the github wiki are for development testing, making the best version of stockfish better. I'm not sure how much it would be appreciated if I inserted levels 1-19 into the pool, and I'm pretty sure the lowest stockfish levels are in the triple digits and I don't think that throwing them in a pool with 3300 rated engines will help me find ratings to the precision I need.
Answers.UserDisplayName: day
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-07T01:07:33.170Z
Answers.Text: I don''t know where to put this. But it appears that "play with computer" on lichess web sites has some associations to its rating system.  I wonder on what basis.  But maybe you already know.  Finding out how they do it, could provide for a way to ELO.  However, there is the question for any paring pool based ratings of defining that pool.
Answers.UserDisplayName: dbdb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-08T03:56:37.910Z
Answers.Text: Sopel already provided the best answer in a comment under the question, but they never actually wrote it up as an answer and stack exhange doesn't allow me to mark just a comment as the correct answer. Lichess engine levels have nothing to do with the actual stockfish setting levels, but they do use the resource sopel linked for estimation.
Answers.UserDisplayName: day
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-07T22:50:25.463Z
Answers.LastActivityDate: 2020-06-07T23:21:59.250Z
Answers.LastEditDate: 2020-06-07T23:21:59.250Z
Answers.UserDisplayName: dbdb
Body: <p>I want to create something that will select an appropriate Stockfish difficulty setting automatically given the player's rating. The straight Stockfish v11 that I've downloaded has 20 difficulty settings, and various other settings for controlling thinking time. Is there a resource that has approximate ratings for the levels? If that doesn't exist, what method would be recommended for finding the approximate ratings of chess engines when I'm starting from scratch?</p>
  <p>EDIT: Sopel gave a <a href="http://github.com/official-stockfish/Stockfish/pull/2225" rel="nofollow noreferrer">link some time ago,</a> that answers the problem, although they did it in comment form so I can't mark it as the correct answer</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-05T10:08:21.943Z
Comments.Text: Not sure how you could really compare a human rating with a computer difficulty level. When you play a computer you pretty much are just playing against yourself. You are playing against your own knowledge and experience. In any given position you are trying to use your own experience to play the best move possible. The level of the computer really only determines how much you will be punished if you don't make the best possible move.
Comments.UserDisplayName: musefan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-05T12:35:54.553Z
Comments.Text: The design of the system I'm building won't really have a good way for the "player" to select the difficulty of the engine, and the "player" will only be able to play against engines. For my purposes, I want to know the rough equivalent ELO of the stockfish levels so that I can calculate adjustments to ELO when the player wins/loses, and so that I can select an appropriate rated engine for the next game.
  
  "Player" in quotation marks because the system is not going to have one human player
Comments.UserDisplayName: day
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-08T22:33:59.150Z
Comments.Text: https://github.com/official-stockfish/Stockfish/pull/2225 is probably the best you'll get
Comments.UserDisplayName: Sopel
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-10T06:33:13.063Z
Comments.Text: Answer from sopel actually looks like the best answer. The files changed section in the link given describes changes to search.cpp, which contains the exact relation. Proving table for the relation given in the summary for peace of mind. Post link as an answer and I'll tick it.
Comments.UserDisplayName: day
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-18T03:24:58.323Z
Comments.Text: ccrl only provides engine strengths for each engine at max strength. Question was looking for a way to find engine strengths of the adjustable levels within stockfish.
Comments.UserDisplayName: day
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-12T11:39:32.043Z
Comments.Text: Here is a list of engine strenght. I think it's quite realistic.
  https://ccrl.chessdom.com/ccrl/4040/
  Hope it helps.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-27T02:16:05.310Z
Comments.Text: I think it will be difficult to account for hardware differences. How can I compare the CCRL ratings with my laptop? I wonder if the skill levels (and their mapping to Elo) could be calibrated using nodes per second.
Comments.UserDisplayName: chesskobra
Tags: <engines><rating><stockfish><programming><elo>
Title: Is there a list of approximate Elo ratings for each Stockfish level?
ThreadText: # Is there a list of approximate Elo ratings for each Stockfish level?
  
  (Asked by: username_0 on 2020-06-04)
  
  I want to create something that will select an appropriate Stockfish difficulty setting automatically given the player's rating. The straight Stockfish v11 that I've downloaded has 20 difficulty settings, and various other settings for controlling thinking time. Is there a resource that has approximate ratings for the levels? If that doesn't exist, what method would be recommended for finding the approximate ratings of chess engines when I'm starting from scratch?
  
  EDIT: username_2 gave a [link some time ago,](http://github.com/official-stockfish/Stockfish/pull/2225) that answers the problem, although they did it in comment form so I can't mark it as the correct answer
  
  **Tags:** engines, rating, stockfish, programming, elo
  
  > **Comments:**
  > **username_1**: Not sure how you could really compare a human rating with a computer difficulty level. When you play a computer you pretty much are just playing against yourself. You are playing against your own knowledge and experience. In any given position you are trying to use your own experience to play the best move possible. The level of the computer really only determines how much you will be punished if you don't make the best possible move.  
  > **username_0**: The design of the system I'm building won't really have a good way for the "player" to select the difficulty of the engine, and the "player" will only be able to play against engines. For my purposes, I want to know the rough equivalent ELO of the stockfish levels so that I can calculate adjustments to ELO when the player wins/loses, and so that I can select an appropriate rated engine for the next game.  "Player" in quotation marks because the system is not going to have one human player  
  > **username_2**: https://github.com/official-stockfish/Stockfish/pull/2225 is probably the best you'll get  
  > **username_0**: Answer from username_2 actually looks like the best answer. The files changed section in the link given describes changes to search.cpp, which contains the exact relation. Proving table for the relation given in the summary for peace of mind. Post link as an answer and I'll tick it.  
  > **username_3**: Here is a list of engine strenght. I think it's quite realistic. https://ccrl.chessdom.com/ccrl/4040/ Hope it helps.  
  > **username_0**: ccrl only provides engine strengths for each engine at max strength. Question was looking for a way to find engine strengths of the adjustable levels within stockfish.  
  > **username_4**: I think it will be difficult to account for hardware differences. How can I compare the CCRL ratings with my laptop? I wonder if the skill levels (and their mapping to Elo) could be calibrated using nodes per second.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_5 on 2022-07-26. Score: 4 | Accepted answer
  
  Stockfish 16 UCI\_Elo was [calibrated](https://github.com/official-stockfish/Stockfish/commit/a08b8d4) with the **CCRL rating pool**. See [Are engine ratings directly comparable to human-level play?](https://chess.stackexchange.com/questions/25328/are-engine-ratings-directly-comparable-to-human-level-play) From the [Stockfish FAQ](https://disservin.github.io/stockfish-docs/stockfish-wiki/Stockfish-FAQ.html#the-elo-rating-of-stockfish): "Finally, rating Stockfish on a human scale (e.g. FIDE Elo) has become an almost impossible task, as strength differences between engines and humans are now so large, that this difference can hardly be measured."
  
  ```
     # PLAYER             :  RATING  ERROR  POINTS  PLAYED   (%)
     1 master-skill-19    :  3191.1   40.4   940.0    1707    55
     2 master-skill-18    :  3170.3   39.3  1343.0    2519    53
     3 master-skill-17    :  3141.3   37.8  2282.0    4422    52
     4 master-skill-16    :  3111.2   37.1  2773.0    5423    51
     5 master-skill-15    :  3069.5   37.2  2728.5    5386    51
     6 master-skill-14    :  3024.8   36.1  2702.0    5339    51
     7 master-skill-13    :  2972.9   35.4  2645.5    5263    50
     8 master-skill-12    :  2923.1   35.0  2653.5    5165    51
     9 master-skill-11    :  2855.5   33.6  2524.0    5081    50
    10 master-skill-10    :  2788.3   32.0  2724.5    5511    49
    11 stash-bot-v25.0    :  2744.0   31.5  1952.5    3840    51
    12 master-skill-9     :  2702.8   30.5  2670.0    5018    53
    13 master-skill-8     :  2596.2   28.5  2669.5    4975    54
    14 stash-bot-v21.0    :  2561.2   30.0  1338.0    3366    40
    15 master-skill-7     :  2499.5   28.5  1934.0    4178    46
    16 stash-bot-v20.0    :  2452.6   27.7  1606.5    3378    48
    17 stash-bot-v19.0    :  2425.3   26.7  1787.0    3365    53
    18 master-skill-6     :  2363.2   26.4  2510.5    4379    57
    19 stash-bot-v17.0    :  2280.7   25.4  2209.0    4378    50
    20 master-skill-5     :  2203.7   25.3  2859.5    5422    53
    21 stash-bot-v15.3    :  2200.0   25.4  1757.0    4383    40
    22 stash-bot-v14      :  2145.9   25.5  2890.0    5167    56
    23 stash-bot-v13      :  2042.7   25.8  2263.5    4363    52
    24 stash-bot-v12      :  1963.4   25.8  1769.5    4210    42
    25 master-skill-4     :  1922.9   25.9  2690.0    5399    50
    26 stash-bot-v11      :  1873.0   26.3  2203.5    4335    51
    27 stash-bot-v10      :  1783.8   27.8  2568.5    4301    60
    28 master-skill-3     :  1742.3   27.8  1909.5    4439    43
    29 master-skill-2     :  1608.4   29.4  2064.5    4389    47
    30 stash-bot-v9       :  1582.6   30.2  2130.0    4230    50
    31 master-skill-1     :  1467.6   31.3  2015.5    4244    47
    32 stash-bot-v8       :  1452.8   31.5  1953.5    3780    52
    33 master-skill-0     :  1320.1   32.9   651.5    2083    31
  
  ```
  
  ---
  
  There was some discussion on improving Stockfish levels to play more humanlike <https://github.com/official-stockfish/Stockfish/issues/3635>
  
  dav1312 provides this ELO rating list, but it's not listed how it was derived:
  
  ```
  0    1347
  1    1490
  2    1597
  3    1694
  4    1785
  5    1871
  6    1954
  7    2035
  8    2113
  9    2189
  10    2264
  11    2337
  12    2409
  13    2480
  14    2550
  15    2619
  16    2686
  17    2754
  18    2820
  19    2886
  
  ```
  
  I think lichess could very well keep track of rating based on all the people that play it like for other bots: <https://lichess.org/player/bots>. I don't think they do, though.
  
  > **Comments:**
  > **username_0**: In the absence of the ability for me to mark username_2's comment as the correct answer because they didn't submit it as an answer, I'll award it to this one as it's very close to the same essence  
  
  ---
  
  ### Answer by username_6 on 2021-01-04. Score: 3
  
  I am rated 1410 USCF. I played all 8 Stockfish levels this past weekend, once with each color. My record was 11-5.
  I was 2-0 vs Stockfish 1-5, 1-1 vs Stockfish 6, and 0-2 vs Stockfish 7 and 8.
  
  > **Comments:**
  > **username_7**: The most concrete answer, based on experience. Only that lacks the reflection time, the hardware were it runs (MHz and MB of RAM) and if pondering was active, to be more complete.  
  > **username_0**: username_2 gave the best answer some months back [here](https://chess.stackexchange.com/questions/29860/is-there-a-list-of-approximate-elo-ratings-for-each-stockfish-level/33613#comment47749_29860) but they did it as a comment that I can't mark as the accepted answer. The lichess stockfish bots only have 8 discrete difficulty options and do not surface where on the actual stockfish difficulty setting scale from 1-20 they correspond to, rendering a winrate ELO analysis of them useless for the purposes of determining strength of a direct interface to the engine.  
  
  ---
  
  ### Answer by username_8 on 2020-06-07. Score: 1
  
  They have a development pipeline doing testing based on one of the early versions.
  I saw a graph of such test in some heavy loading page from Next Chess Move website:
  
  * [Stockfish Development Builds](https://nextchessmove.com/dev-builds)
  
  The development page gathering the test linked from the Fishtest wiki:
  
  * [Stockfish Testing Queue](https://tests.stockfishchess.org/tests)
  
  And for completion:
  
  * [Fishtest GitHub Repository Wiki](https://github.com/glinscott/fishtest/wiki)
  
  > **Comments:**
  > **username_0**: These resources seem to be for evaluating optimisations to the engine in development, I don't see any tests evaluating the engine set to levels 1-19, not sure that fishtest would be receptive to having levels 1-19 put in the system given their stated goals  
  > **username_8**: I have not looked into the details, but if I were testing for possible regression or improvements, i would use the whole dynamic range available (both sides), I would expect that somewhere is that mass of data there should be a way to get some approximate ELO relationship to the parameter volume explored.  There was no answer.  I hope there will be a better answer later on.  I am also curious.  Do you know the stated goals?  
  > **username_0**: The results in the first two links don't seem to have any examples of stockfish set to any level besides the highest one. The stated goals of fishtest by the github wiki are for development testing, making the best version of stockfish better. I'm not sure how much it would be appreciated if I inserted levels 1-19 into the pool, and I'm pretty sure the lowest stockfish levels are in the triple digits and I don't think that throwing them in a pool with 3300 rated engines will help me find ratings to the precision I need.  
  > **username_8**: I don''t know where to put this. But it appears that "play with computer" on lichess web sites has some associations to its rating system.  I wonder on what basis.  But maybe you already know.  Finding out how they do it, could provide for a way to ELO.  However, there is the question for any paring pool based ratings of defining that pool.  
  > **username_0**: username_2 already provided the best answer in a comment under the question, but they never actually wrote it up as an answer and stack exhange doesn't allow me to mark just a comment as the correct answer. Lichess engine levels have nothing to do with the actual stockfish setting levels, but they do use the resource username_2 linked for estimation.  
  
  ---
  
  