Answers.Body: <p>There're a few ways to do it:</p>
  <ul>
  <li>Add 400 rating to the top opponent and take that as an approximation (used by <a href="https://www.uschess.org/archive/ratings/calculator.html" rel="nofollow noreferrer">USCF</a>)</li>
  <li>Add a draw for the engine against itself</li>
  <li>Assign an infinite rating</li>
  <li><a href="http://chess.kivij.info/performance_calculator.shtml" rel="nofollow noreferrer">Use linear approximation</a></li>
  </ul>
  <p>By the way &quot;win every game against the most powerful engines and players&quot; is realistically not happening, because <a href="https://en.wikipedia.org/wiki/First-move_advantage_in_chess#Drawn_with_best_play" rel="nofollow noreferrer">chess is a draw</a>** and top engines on strong hardware guided by a human are already so close to perfect play that the great majority of all correspondence chess games are drawn.</p>
  <p>**It's not proven that chess is a draw, but you won't find any top player seriously disagreeing with this assessment. See also <a href="https://chess.stackexchange.com/questions/5158/are-world-class-correspondence-games-more-likely-to-be-drawn-than-world-class-ot/31574#31574">this answer</a> illustrating what things are like at the top level of play (world correspondence chess championship, which is stronger than both engines and humans alone).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-04T13:16:24.530Z
Answers.Text: We cannot know if (current engine + human) are close to perfect play, right?  "Perfect play" is usually a term we define based on what the engine says, so with a drastically improved engine, it's entirely possible that the new engine would win every game.
Answers.UserDisplayName: Nathan Merrill
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-04T13:39:19.587Z
Answers.Text: "Chess is a draw" is not a mathematically precise argument. In a hypothetical scenario, if the game of chess is solved then the side equipped with the winning algorithm will have the ultimate power assumed in the question.
Answers.UserDisplayName: polfosol
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-04T13:41:26.567Z
Answers.Text: "The side with the winning algorithm" doesn't exist if perfect play leads to a draw, though. And current indications are that, once solved, chess will be in that class.
Answers.UserDisplayName: L. Scott Johnson
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-04T13:41:48.330Z
Answers.Text: @polfosol: "Chess is a draw" is precise. That's not the problem. Game Theory says the exact same thing for Tic Tac Toe. The difference is that for Tic Tac Toe, there's proof. For chess, it's an unproven assertion, but it's an unambiguous assertion
Answers.UserDisplayName: MSalters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-05T03:02:50.783Z
Answers.Text: @NathanMerrill engines are at the point where bookless games (i.e. games played from the opening position) on strong hardware at long time controls are almost always drawn. See e.g. this game: White (Leela) has a 300-elo rating advantage and is playing against an opponent with 30% of their time, and the game is still drawn. https://tcec-chess.com/#div=sft&game=12&season=21
Answers.UserDisplayName: Allure
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-05T04:28:58.550Z
Answers.Text: @Allure ah interesting.  So because draws still occur despite large elo differences, we'd expect that to continue with even better engines. That's pretty convincing, even if it isn't a proof.
Answers.UserDisplayName: Nathan Merrill
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-04T02:03:20.053Z
Answers.LastActivityDate: 2022-01-05T11:27:51.260Z
Answers.LastEditDate: 2022-01-05T11:27:51.260Z
Answers.UserDisplayName: Allure
Answers.Body: <p>According to <a href="https://www.remi-coulom.fr/Bayesian-Elo/" rel="nofollow noreferrer">bayeselo</a> program if the best wins 1k games without loses and draws and setting the opponent rating to 1000, it will get 2092.</p>
  <pre><code>Rank Name:   Elo    +    - games score oppo. draws   win  loss  draw
     1 best:  2092  170   83  1000  100%  1000    0%  1000     0     0
     2 p1  :  1000   83  170  1000    0%  2092    0%     0  1000     0
  </code></pre>
  <p>1M games:</p>
  <pre><code>Rank Name:   Elo    +    - games score oppo. draws   win  loss  draw
     1 best:  3203  167   82 1e+06  100%  1000    0% 1e+06     0     0
     2 p1  :  1000   82  167 1e+06    0%  3203    0%     0 1e+06     0
  </code></pre>
  <p>2M games:</p>
  <pre><code>Rank Name:   Elo    +    - games score oppo. draws   win  loss  draw
     1 best:  3314  161   83 2e+06  100%  1000    0% 2e+06     0     0
     2 p1  :  1000   83  161 2e+06    0%  3314    0%     0 2e+06     0
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-04T08:05:12.670Z
Answers.LastActivityDate: 2022-01-04T08:05:12.670Z
Answers.UserDisplayName: ferdy
Answers.Body: <p>While there will be no concrete answer to this, there are few methods of approximation. We can play it against a 7 man tablebase, and for the sake of ELO rating we can assume that its strength in the middlegame or the opening is the same as the endgame. The number of losses that an endgame tablebase delivers to it (giving the borderline winning side to the tablebase and a blessed loss to the engine) determines its strength.</p>
  <p>Another method we can use is the number of moves it takes to defeat the opposing engine. While not always correct, the general rule is that the more moves it takes to defeat the opponent, the weaker the player.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-04T13:13:09.153Z
Answers.Text: Presumably the super-strong engine is itself using an endgame tablebase internally. The assumption *"We can play it against a 7 man tablebase, and for the sake of ELO rating we can assume that its strength in the middlegame or the opening is the same as the endgame."* is inherently flawed, because the endgame is a solved problem, whereas the opening is an open problem.
Answers.UserDisplayName: Stef
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-04T19:44:05.297Z
Answers.Text: If the source code is known for the super strong engine, then you don't need to approximate at all. You can modify it to be weaker and play it against its weaker version. Keep on doing that and creating weaker versions until they match the next strongest engine.
Answers.UserDisplayName: birdo39
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-04T01:19:05.680Z
Answers.LastActivityDate: 2022-01-04T01:19:05.680Z
Answers.UserDisplayName: birdo39
Answers.Body: <p>You could let the engine play with a handicap (e.g. rook down).</p>
  <p>Then, you need to calibrate how much rating delta a missing rook means. I saw some numbers that estimated one point of material to be a delta of 100 points.</p>
  <p>This calibration could be done through self-play. Let the engine play against itself and see how much a rook is worth.</p>
  <p>A downside of this handicap method is that it changes the game dynamics. Engines are not optimized for starting the game in an unusual position with a massive disadvantage. This can throw off some heuristics.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-04T11:49:05.470Z
Answers.LastActivityDate: 2022-01-04T11:49:05.470Z
Answers.UserDisplayName: usr
Answers.Body: <p>I'd treat this as a Bayesian inverse probability problem. Per Laplace's rule-of-succession solution of the sunrise problem (see <a href="https://en.wikipedia.org/wiki/Rule_of_succession" rel="noreferrer">https://en.wikipedia.org/wiki/Rule_of_succession</a> and <a href="https://en.wikipedia.org/wiki/Sunrise_problem" rel="noreferrer">https://en.wikipedia.org/wiki/Sunrise_problem</a>), it follows that <strong>if an engine wins <em>n</em> out of <em>n</em> games, its rating should be estimated as if it played <em>n+3</em> games, winning <em>n+1</em>, drawing one, and losing one</strong> - playing, of course, against the same pool of opponents. (A point of note: a chess game differs from a sunrise in that there are <em>three</em> possible outcomes, rather than just two. For the sake of simplicity, we'll assume the number of games with black and white pieces is equal.)</p>
  <p>Using the example provided above, the rule-of-succession method returns an estimate of 2230 (1000 games), 3330 (1M games), and 3450 (2M games) - not that far off from Bayeselo. I'm not familiar with the method employed by Bayeselo and cannot say how mathematically sound it is, but in the end this is a kind of problem that doesn't have a &quot;correct&quot; solution because it hinges on one's choice of prior probability. In Laplace's approach, it is posited that, if we knew nothing about the history of sunrises, we should assume that the probability of Sun rising tomorrow is uniformly distributed between 0 and 1. Whether that assumption is applicable to a chess game between two players about whom we know nothing is anyone's guess.</p>
  <p>(Additional note: I'm assuming here that the question is how to estimate Elo once <em>n</em> games have already been played, rather than when to stop playing more games. The answer to the latter question is significantly more complex as it obviously depends on additional factors.)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-04T12:12:52.743Z
Answers.LastActivityDate: 2022-01-04T12:12:52.743Z
Answers.UserDisplayName: Branko Radovanovic
Body: <p>Suppose we have an extremely strong engine that happens to win every game it participates in even against the most powerful engines and players.
  If it continues to win every game, its rating (for example ELO) will grow indefinitely. But we do not have an infinite amount of time, so we will have to stop evaluating sometimes with some rating R.
  On the other hand, it feels wrong to stop at rating R because it is likely our engine will win its next game and get a rating R' &gt; R.
  Moreover, we cannot give this engine a rating of infinity, because in theory it can draw or lose, it just hasn't yet faced such an opponent.</p>
  <p>How would you find a rating approximation for such an engine?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-04T13:08:22.757Z
Comments.Text: Similar, much simpler problem from statistics: you have a biased coin with an unknown parameter *p*, that lands on Heads with proba *p* and on Tails with proba *1-p*. You want to estimate *p* by flipping the coin repeatedly. If *p* is close to 50%, then this is quickly going to give a very good estimation. But if *p* is close to 0 or to 1, ie if the coin almost-always lands on Heads or almost-always lands on tails, then estimating *p* precisely is going to be very difficult.
Comments.UserDisplayName: Stef
Tags: <engines><rating><mathematics>
Title: Approximating a rating for too strong engines
ThreadText: # Approximating a rating for too strong engines
  
  (Asked by: username_0 on 2022-01-04)
  
  Suppose we have an extremely strong engine that happens to win every game it participates in even against the most powerful engines and players.
  If it continues to win every game, its rating (for example ELO) will grow indefinitely. But we do not have an infinite amount of time, so we will have to stop evaluating sometimes with some rating R.
  On the other hand, it feels wrong to stop at rating R because it is likely our engine will win its next game and get a rating R' > R.
  Moreover, we cannot give this engine a rating of infinity, because in theory it can draw or lose, it just hasn't yet faced such an opponent.
  
  How would you find a rating approximation for such an engine?
  
  **Tags:** engines, rating, mathematics
  
  > **Comments:**
  > **username_1**: Similar, much simpler problem from statistics: you have a biased coin with an unknown parameter *p*, that lands on Heads with proba *p* and on Tails with proba *1-p*. You want to estimate *p* by flipping the coin repeatedly. If *p* is close to 50%, then this is quickly going to give a very good estimation. But if *p* is close to 0 or to 1, ie if the coin almost-always lands on Heads or almost-always lands on tails, then estimating *p* precisely is going to be very difficult.  
  
  ---
  
  ## 5 answers
  
  ### Answer by username_10 on 2022-01-04. Score: 7 | Accepted answer
  
  I'd treat this as a Bayesian inverse probability problem. Per Laplace's rule-of-succession solution of the sunrise problem (see <https://en.wikipedia.org/wiki/Rule_of_succession> and <https://en.wikipedia.org/wiki/Sunrise_problem>), it follows that **if an engine wins *n* out of *n* games, its rating should be estimated as if it played *n+3* games, winning *n+1*, drawing one, and losing one** - playing, of course, against the same pool of opponents. (A point of note: a chess game differs from a sunrise in that there are *three* possible outcomes, rather than just two. For the sake of simplicity, we'll assume the number of games with black and white pieces is equal.)
  
  Using the example provided above, the rule-of-succession method returns an estimate of 2230 (1000 games), 3330 (1M games), and 3450 (2M games) - not that far off from Bayeselo. I'm not familiar with the method employed by Bayeselo and cannot say how mathematically sound it is, but in the end this is a kind of problem that doesn't have a "correct" solution because it hinges on one's choice of prior probability. In Laplace's approach, it is posited that, if we knew nothing about the history of sunrises, we should assume that the probability of Sun rising tomorrow is uniformly distributed between 0 and 1. Whether that assumption is applicable to a chess game between two players about whom we know nothing is anyone's guess.
  
  (Additional note: I'm assuming here that the question is how to estimate Elo once *n* games have already been played, rather than when to stop playing more games. The answer to the latter question is significantly more complex as it obviously depends on additional factors.)
  
  
  ---
  
  ### Answer by username_2 on 2022-01-04. Score: 11
  
  There're a few ways to do it:
  
  * Add 400 rating to the top opponent and take that as an approximation (used by [USCF](https://www.uschess.org/archive/ratings/calculator.html))
  * Add a draw for the engine against itself
  * Assign an infinite rating
  * [Use linear approximation](http://chess.kivij.info/performance_calculator.shtml)
  
  By the way "win every game against the most powerful engines and players" is realistically not happening, because [chess is a draw](https://en.wikipedia.org/wiki/First-move_advantage_in_chess#Drawn_with_best_play)\*\* and top engines on strong hardware guided by a human are already so close to perfect play that the great majority of all correspondence chess games are drawn.
  
  \*\*It's not proven that chess is a draw, but you won't find any top player seriously disagreeing with this assessment. See also [this answer](https://chess.stackexchange.com/questions/5158/are-world-class-correspondence-games-more-likely-to-be-drawn-than-world-class-ot/31574#31574) illustrating what things are like at the top level of play (world correspondence chess championship, which is stronger than both engines and humans alone).
  
  > **Comments:**
  > **username_3**: We cannot know if (current engine + human) are close to perfect play, right?  "Perfect play" is usually a term we define based on what the engine says, so with a drastically improved engine, it's entirely possible that the new engine would win every game.  
  > **username_4**: "Chess is a draw" is not a mathematically precise argument. In a hypothetical scenario, if the game of chess is solved then the side equipped with the winning algorithm will have the ultimate power assumed in the question.  
  > **username_5**: "The side with the winning algorithm" doesn't exist if perfect play leads to a draw, though. And current indications are that, once solved, chess will be in that class.  
  > **username_6**: @username_4: "Chess is a draw" is precise. That's not the problem. Game Theory says the exact same thing for Tic Tac Toe. The difference is that for Tic Tac Toe, there's proof. For chess, it's an unproven assertion, but it's an unambiguous assertion  
  > **username_2**: @username_3 engines are at the point where bookless games (i.e. games played from the opening position) on strong hardware at long time controls are almost always drawn. See e.g. this game: White (Leela) has a 300-elo rating advantage and is playing against an opponent with 30% of their time, and the game is still drawn. https://tcec-chess.com/#div=sft&game=12&season=21  
  > **username_3**: @username_2 ah interesting.  So because draws still occur despite large elo differences, we'd expect that to continue with even better engines. That's pretty convincing, even if it isn't a proof.  
  
  ---
  
  ### Answer by username_7 on 2022-01-04. Score: 2
  
  According to [bayeselo](https://www.remi-coulom.fr/Bayesian-Elo/) program if the best wins 1k games without loses and draws and setting the opponent rating to 1000, it will get 2092.
  
  ```
  Rank Name:   Elo    +    - games score oppo. draws   win  loss  draw
     1 best:  2092  170   83  1000  100%  1000    0%  1000     0     0
     2 p1  :  1000   83  170  1000    0%  2092    0%     0  1000     0
  
  ```
  
  1M games:
  
  ```
  Rank Name:   Elo    +    - games score oppo. draws   win  loss  draw
     1 best:  3203  167   82 1e+06  100%  1000    0% 1e+06     0     0
     2 p1  :  1000   82  167 1e+06    0%  3203    0%     0 1e+06     0
  
  ```
  
  2M games:
  
  ```
  Rank Name:   Elo    +    - games score oppo. draws   win  loss  draw
     1 best:  3314  161   83 2e+06  100%  1000    0% 2e+06     0     0
     2 p1  :  1000   83  161 2e+06    0%  3314    0%     0 2e+06     0
  
  ```
  
  
  ---
  
  ### Answer by username_8 on 2022-01-04. Score: 2
  
  While there will be no concrete answer to this, there are few methods of approximation. We can play it against a 7 man tablebase, and for the sake of ELO rating we can assume that its strength in the middlegame or the opening is the same as the endgame. The number of losses that an endgame tablebase delivers to it (giving the borderline winning side to the tablebase and a blessed loss to the engine) determines its strength.
  
  Another method we can use is the number of moves it takes to defeat the opposing engine. While not always correct, the general rule is that the more moves it takes to defeat the opponent, the weaker the player.
  
  > **Comments:**
  > **username_1**: Presumably the super-strong engine is itself using an endgame tablebase internally. The assumption *"We can play it against a 7 man tablebase, and for the sake of ELO rating we can assume that its strength in the middlegame or the opening is the same as the endgame."* is inherently flawed, because the endgame is a solved problem, whereas the opening is an open problem.  
  > **username_8**: If the source code is known for the super strong engine, then you don't need to approximate at all. You can modify it to be weaker and play it against its weaker version. Keep on doing that and creating weaker versions until they match the next strongest engine.  
  
  ---
  
  ### Answer by username_9 on 2022-01-04. Score: 0
  
  You could let the engine play with a handicap (e.g. rook down).
  
  Then, you need to calibrate how much rating delta a missing rook means. I saw some numbers that estimated one point of material to be a delta of 100 points.
  
  This calibration could be done through self-play. Let the engine play against itself and see how much a rook is worth.
  
  A downside of this handicap method is that it changes the game dynamics. Engines are not optimized for starting the game in an unusual position with a massive disadvantage. This can throw off some heuristics.
  
  
  ---
  
  