Answers.Body: <p>Elo calculations are not a statistical estimate of an unknown parameter, they are just an aggregate count of actual results. Since all those results are known, the &quot;error bar&quot; would have size 0.</p>
  <p>It is possible to perform a test of statistical significance over match results to determine if there's enough evidence to conclude that one engine/player is stronger than the other, but that's a separate problem from Elo computation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-14T09:07:28.070Z
Answers.Text: *It is possible to perform a test of statistical significance over the match results to determine if there's enough evidence to conclude that one engine/player is stronger than the other* Do you know how to do this test, e.g. with the match in the OP as an example?
Answers.UserDisplayName: Allure
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-14T09:12:32.633Z
Answers.Text: @Allure with the information we have the best choice would probably be a proportions test (https://www.statology.org/one-proportion-z-test/). Let p be the average score Stockfish would get against Leela and let's take p=0.5 as the null hypothesis. However since chess is a game with three possible results, it'd be more accurate to have the win/draw/loss count rather than just the final score. Then we can check if the number of wins is significantly bigger than the number of losses instead of just a test over the final score.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-15T01:11:49.407Z
Answers.Text: @David To get to the spirit of the question, would it be valid to just go to https://www.statology.org/one-proportion-z-test-calculator/ and fill in p=0.505 and n=100, look at the 95% confidence intervals, and plug them into the FIDE fractional-score-to-rating table to convert them to a ratings difference? Or does the 3 result possibility throw this calculation off? (If I do that, I get 95% C.I. = [0.4070, 0.6030] which gives me about -67 to +74.)
Answers.UserDisplayName: D M
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-15T11:08:29.700Z
Answers.Text: @DM In this case it'd be the same since even if it was 1 win and 99 draws you wouldn't get a significant difference. The proportions test approximation is good enough. One could also try a t-test for normal distributions as a reasonable approximation
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-14T08:56:56.080Z
Answers.LastActivityDate: 2023-04-21T06:09:15.810Z
Answers.LastEditDate: 2023-04-21T06:09:15.810Z
Answers.UserDisplayName: David
Answers.Body: <p>Here is a practical answer: Since, as already said, the calculation is &quot;exact&quot;, look instead at the rating fluctuations of players since by any model, the rating of a (medium aged) player is constant, modulo local form fluctuations, aging and whatnot. These random fluctuations should be a far better proxy for the &quot;real&quot; sigma of the Elo model than any contrieved math. (Only don't take <em>my</em> charts :-)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-21T08:05:59.590Z
Answers.LastActivityDate: 2023-04-21T08:05:59.590Z
Answers.UserDisplayName: Hauke Reddmann
Body: <p>In season 14 of TCEC, Stockfish defeated Leela 50.5-49.5 in a 100-game match (the closest possible winning margin). Using an elo calculator such as <a href="https://www.3dkingdoms.com/chess/elo.htm" rel="nofollow noreferrer">this one</a>, we can see that the elo difference between the two engines was +3.</p>
  <p>However it must be that this isn't a statistically significant result - after all the match was only 100 games, and it's the smallest possible winning margin. Therefore the error bars on this calculation should be at least 3. The calculator doesn't give the error bars though, and I can't find any equations on how to calculate the error. How do you calculate the error on this estimate?</p>
  <p>Related: <a href="https://chess.stackexchange.com/questions/15135/how-do-you-calculate-elo">How do you calculate elo?</a></p>
  <p><strong>Edit</strong>: Looks like 3dkingdoms have improved their calculator, which is now capable of calculating an error margin. I'd still be interested in seeing how that error margin is calculated, though.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-23T09:09:24.120Z
Comments.Text: I'm voting to close this question as off-topic because it is a maths question, not a chess question.
Comments.UserDisplayName: Brian Towers
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-23T20:38:34.743Z
Comments.Text: LOL. People ask questions about shogi and things like that all the time
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-23T21:29:55.593Z
Comments.Text: I'd put this on Math.SE, but then first I'd have to explain what elo is, how it translates to expected performance, etc.
Comments.UserDisplayName: Allure
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-13T06:06:39.500Z
Comments.Text: @David are you saying it should be re-opened?
Comments.UserDisplayName: BCLC
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-13T13:08:14.107Z
Comments.Text: @Allure I think [CrossValidated](https://stats.stackexchange.com/) (Stats.SE) can handle this if Chess.SE won't allow it.  You could probably reword it to "What is the uncertainty on the rating estimate in the Elo model? in order to make it suitable here.
Comments.UserDisplayName: SecretAgentMan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-13T16:20:24.163Z
Comments.Text: Voting to reopen. If the math involved here isn't sufficiently chess related, then neither is almost any other "chess question".
Comments.UserDisplayName: Charles Rockafellor
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-14T14:57:30.060Z
Comments.Text: The error bars for an Elo estimate depend on the number of games played and how well-matched the players are. The more games played and the greater the difference in skill between the players, the smaller the error bars. There isn't a set formula to calculate the error bars, but statistical methods like the binomial distribution can be used to get an estimate. The Elo rating system is based on a player's performance against other players and gets updated after each game based on the result and the player's rating compared to their opponent's.
Comments.UserDisplayName: Error404
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-16T17:47:29.477Z
Comments.Text: ELO hasn't error bars by definition. Its calculated over results. For hipotetical problems generate 1000 aleatorial results in a spreadsheet using the Elo formula.
Comments.UserDisplayName: djnavas
Tags: <rating>
Title: How do you calculate error bars on elo rating?
ThreadText: # How do you calculate error bars on elo rating?
  
  (Asked by: username_0 on 2019-05-22)
  
  In season 14 of TCEC, Stockfish defeated Leela 50.5-49.5 in a 100-game match (the closest possible winning margin). Using an elo calculator such as [this one](https://www.3dkingdoms.com/chess/elo.htm), we can see that the elo difference between the two engines was +3.
  
  However it must be that this isn't a statistically significant result - after all the match was only 100 games, and it's the smallest possible winning margin. Therefore the error bars on this calculation should be at least 3. The calculator doesn't give the error bars though, and I can't find any equations on how to calculate the error. How do you calculate the error on this estimate?
  
  Related: [How do you calculate elo?](https://chess.stackexchange.com/questions/15135/how-do-you-calculate-elo)
  
  **Edit**: Looks like 3dkingdoms have improved their calculator, which is now capable of calculating an error margin. I'd still be interested in seeing how that error margin is calculated, though.
  
  **Tags:** rating
  
  > **Comments:**
  > **username_1**: I'm voting to close this question as off-topic because it is a maths question, not a chess question.  
  > **username_2**: LOL. People ask questions about shogi and things like that all the time  
  > **username_0**: I'd put this on Math.SE, but then first I'd have to explain what elo is, how it translates to expected performance, etc.  
  > **username_3**: @username_2 are you saying it should be re-opened?  
  > **username_4**: @username_0 I think [CrossValidated](https://stats.stackexchange.com/) (Stats.SE) can handle this if Chess.SE won't allow it.  You could probably reword it to "What is the uncertainty on the rating estimate in the Elo model? in order to make it suitable here.  
  > **username_5**: Voting to reopen. If the math involved here isn't sufficiently chess related, then neither is almost any other "chess question".  
  > **username_6**: The error bars for an Elo estimate depend on the number of games played and how well-matched the players are. The more games played and the greater the difference in skill between the players, the smaller the error bars. There isn't a set formula to calculate the error bars, but statistical methods like the binomial distribution can be used to get an estimate. The Elo rating system is based on a player's performance against other players and gets updated after each game based on the result and the player's rating compared to their opponent's.  
  > **username_7**: ELO hasn't error bars by definition. Its calculated over results. For hipotetical problems generate 1000 aleatorial results in a spreadsheet using the Elo formula.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-04-14. Score: 1
  
  Elo calculations are not a statistical estimate of an unknown parameter, they are just an aggregate count of actual results. Since all those results are known, the "error bar" would have size 0.
  
  It is possible to perform a test of statistical significance over match results to determine if there's enough evidence to conclude that one engine/player is stronger than the other, but that's a separate problem from Elo computation.
  
  > **Comments:**
  > **username_0**: *It is possible to perform a test of statistical significance over the match results to determine if there's enough evidence to conclude that one engine/player is stronger than the other* Do you know how to do this test, e.g. with the match in the OP as an example?  
  > **username_2**: @username_0 with the information we have the best choice would probably be a proportions test (https://www.statology.org/one-proportion-z-test/). Let p be the average score Stockfish would get against Leela and let's take p=0.5 as the null hypothesis. However since chess is a game with three possible results, it'd be more accurate to have the win/draw/loss count rather than just the final score. Then we can check if the number of wins is significantly bigger than the number of losses instead of just a test over the final score.  
  > **username_8**: @username_2 To get to the spirit of the question, would it be valid to just go to https://www.statology.org/one-proportion-z-test-calculator/ and fill in p=0.505 and n=100, look at the 95% confidence intervals, and plug them into the FIDE fractional-score-to-rating table to convert them to a ratings difference? Or does the 3 result possibility throw this calculation off? (If I do that, I get 95% C.I. = [0.4070, 0.6030] which gives me about -67 to +74.)  
  > **username_2**: @DM In this case it'd be the same since even if it was 1 win and 99 draws you wouldn't get a significant difference. The proportions test approximation is good enough. One could also try a t-test for normal distributions as a reasonable approximation  
  
  ---
  
  ### Answer by username_9 on 2023-04-21. Score: 0
  
  Here is a practical answer: Since, as already said, the calculation is "exact", look instead at the rating fluctuations of players since by any model, the rating of a (medium aged) player is constant, modulo local form fluctuations, aging and whatnot. These random fluctuations should be a far better proxy for the "real" sigma of the Elo model than any contrieved math. (Only don't take *my* charts :-)
  
  
  ---
  
  