Answers.Body: <p>I also try to take part in that competition, although I'm not sure I will be able to spare the time and computational effort to actually make a submission. </p>
  
  <p>Unfortunately I think the Kaggle data set of engine evaluations isn't suitable to get the best possible result. I think it is crucial to not only have the evaluation of the best move, but also of the inferior choices. 
  The reason is that you need to calculate some kind of measure of the complexity of the position to weight the accuracy shown by the players. </p>
  
  <p>In short: It is hard to be accurate in a complex position and easy to be accurate in a simple position.</p>
  
  <p>Another problem is that at a runtime of one second per move, stockfish probably isn't much stronger than the average player in the dataset (which would be around 2200 Elo) …</p>
  
  <p>So to create a decent dataset one would have to invest a lot of cpu-hours. 50000 games x 40 moves x 1second (or more) >= 555 hours.</p>
  
  <p>To create an opening tree that allows you to see when the players deviate from already played games and how strong the players of the played games were, is another idea to give an initial guess of the players strength.</p>
  
  <p>So my rough outline is to create an opening tree to get an initial guess of the strength, ignore the moves played in the opening, weight the accuracy of the moves after the opening phase with the complexity of the position and do some kind of linear regression analysis. Maybe look out not to overvalue time trouble blunders.</p>
  
  <p>I think the key is to get a good "complexity number". The other possibilities you mention … "number of blunders" just reduces "accuracy of play" to something similar with less information. "Depth of tactical traps avoided" is tricky and may computationally be even more expensive.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2015-01-07T10:05:21.560Z
Answers.LastActivityDate: 2015-01-07T10:05:21.560Z
Answers.UserDisplayName: BlindKungFuMaster
Body: <p>I'm basically fishing for suggestions here... there is <a href="http://www.kaggle.com/c/finding-elo" rel="nofollow">an AI contest to figure out a person's rating from a single game</a>. The contest's description mentions Ken Regan's approach that makes use of engine evaluations of players' moves (and the contest's accompanying game data is pre-equipped with engine evaluations for such use). What other kinds of factors do you believe would be useful for such a task?</p>
  
  <p>I could imagine things like:</p>
  
  <ol>
  <li>Ability to follow recent opening theory (for strong players)</li>
  <li>Total number of blunders (for weak players)</li>
  <li>At what point in the game did resignation occur? (for strong players, it often evaluates to numbers closer to a draw than for weak players)</li>
  <li>Average depth of tactical traps avoided</li>
  </ol>
  
  <p>Other ideas?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2015-01-07T00:30:01.753Z
Comments.Text: possible duplicate of [Is there a software to measure a player's strength based on a game?](http://chess.stackexchange.com/questions/5656/is-there-a-software-to-measure-a-players-strength-based-on-a-game)
Comments.UserDisplayName: Dag Oskar Madsen
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2015-01-07T00:36:17.633Z
Comments.Text: Short answer: You can‘t.
Comments.UserDisplayName: Keba
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2015-01-07T00:57:00.860Z
Comments.Text: So I assume there would be a human with a known rating that is kept secret. The AIs would play a game then estimate the human's rating?  Or, two players with secret ratings would play, and the AIs would attempt to estimate each??
Comments.UserDisplayName: Tony Ennis
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2015-01-07T02:22:47.810Z
Comments.Text: @DagOskarMadsen, while I agree that this question (especially with its title) looks very much like that earlier one in many respects, I don't believe it is a duplicate. In particular, the earlier question asked for software that could do such a thing, and its accepted answer was about Ken Regan's algorithm based on engine evaluations. Here, tbischel's question links to a kaggle competition whose description already mentions Regan's approach and whose dataset comes equipped with engine evaluations for all moves. (cont'd)
Comments.UserDisplayName: ETD
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2015-01-07T02:24:43.520Z
Comments.Text: (cont'd) I see the present question as essentially asking how one can go beyond Regan's work, e.g. by complementing its approach with additional considerations, in order to do a better job of predicting Elo ratings. Perhaps it could use some rewording, but at least as I read it, it isn't a duplicate. (@tbischel, I have edited the question along the lines I described, but if I have altered your intent at all, please roll it back.)
Comments.UserDisplayName: ETD
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2015-01-07T09:21:31.890Z
Comments.Text: @ETD I see the previous question as essentially asking how you can do a good job of predicting Elo ratings, of which this is a duplicate. If the answer to the earlier question is not good enough, is it a good enough reason to ask the same question again?
Comments.UserDisplayName: JiK
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2015-01-07T09:26:20.333Z
Comments.Text: I'd also vote to close this question because it is primarily opinion based and subjective. If there is a good objective answer to the question, it has to have the required background research and cite the essential sources. But then that method is already going to be used in the Kaggle competition, and thus it is not an answer to this question.
Comments.UserDisplayName: JiK
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2015-01-12T23:00:01.213Z
Comments.Text: @Jik "If there is a good objective answer to the question, it has to have the required background research and cite the essential sources." Yes, or it could be an original contribution of such work. "But then that method is already going to be used in the Kaggle competition, and thus it is not an answer to this question." I don't see why something being able to be *used* in the competition would make it not an answer here. It's only the fact that Regan's approach is mentioned already in the *posing* of the referenced competition that I see this Q as asking for approaches beyond that.
Comments.UserDisplayName: ETD
Tags: <elo>
Title: How would you figure out someone's Elo rating from a single game of chess?
ThreadText: # How would you figure out someone's Elo rating from a single game of chess?
  
  (Asked by: username_0 on 2015-01-07)
  
  I'm basically fishing for suggestions here... there is [an AI contest to figure out a person's rating from a single game](http://www.kaggle.com/c/finding-elo). The contest's description mentions Ken Regan's approach that makes use of engine evaluations of players' moves (and the contest's accompanying game data is pre-equipped with engine evaluations for such use). What other kinds of factors do you believe would be useful for such a task?
  
  I could imagine things like:
  
  1. Ability to follow recent opening theory (for strong players)
  2. Total number of blunders (for weak players)
  3. At what point in the game did resignation occur? (for strong players, it often evaluates to numbers closer to a draw than for weak players)
  4. Average depth of tactical traps avoided
  
  Other ideas?
  
  **Tags:** elo
  
  > **Comments:**
  > **username_1**: possible duplicate of [Is there a software to measure a player's strength based on a game?](http://chess.stackexchange.com/questions/5656/is-there-a-software-to-measure-a-players-strength-based-on-a-game)  
  > **username_2**: Short answer: You can‘t.  
  > **username_3**: So I assume there would be a human with a known rating that is kept secret. The AIs would play a game then estimate the human's rating?  Or, two players with secret ratings would play, and the AIs would attempt to estimate each??  
  > **username_4**: @username_1, while I agree that this question (especially with its title) looks very much like that earlier one in many respects, I don't believe it is a duplicate. In particular, the earlier question asked for software that could do such a thing, and its accepted answer was about Ken Regan's algorithm based on engine evaluations. Here, username_0's question links to a kaggle competition whose description already mentions Regan's approach and whose dataset comes equipped with engine evaluations for all moves. (cont'd)  
  > **username_4**: (cont'd) I see the present question as essentially asking how one can go beyond Regan's work, e.g. by complementing its approach with additional considerations, in order to do a better job of predicting Elo ratings. Perhaps it could use some rewording, but at least as I read it, it isn't a duplicate. (@username_0, I have edited the question along the lines I described, but if I have altered your intent at all, please roll it back.)  
  > **username_5**: @username_4 I see the previous question as essentially asking how you can do a good job of predicting Elo ratings, of which this is a duplicate. If the answer to the earlier question is not good enough, is it a good enough reason to ask the same question again?  
  > **username_5**: I'd also vote to close this question because it is primarily opinion based and subjective. If there is a good objective answer to the question, it has to have the required background research and cite the essential sources. But then that method is already going to be used in the Kaggle competition, and thus it is not an answer to this question.  
  > **username_4**: @Jik "If there is a good objective answer to the question, it has to have the required background research and cite the essential sources." Yes, or it could be an original contribution of such work. "But then that method is already going to be used in the Kaggle competition, and thus it is not an answer to this question." I don't see why something being able to be *used* in the competition would make it not an answer here. It's only the fact that Regan's approach is mentioned already in the *posing* of the referenced competition that I see this Q as asking for approaches beyond that.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_6 on 2015-01-07. Score: 3
  
  I also try to take part in that competition, although I'm not sure I will be able to spare the time and computational effort to actually make a submission.
  
  Unfortunately I think the Kaggle data set of engine evaluations isn't suitable to get the best possible result. I think it is crucial to not only have the evaluation of the best move, but also of the inferior choices.
  The reason is that you need to calculate some kind of measure of the complexity of the position to weight the accuracy shown by the players.
  
  In short: It is hard to be accurate in a complex position and easy to be accurate in a simple position.
  
  Another problem is that at a runtime of one second per move, stockfish probably isn't much stronger than the average player in the dataset (which would be around 2200 Elo) …
  
  So to create a decent dataset one would have to invest a lot of cpu-hours. 50000 games x 40 moves x 1second (or more) >= 555 hours.
  
  To create an opening tree that allows you to see when the players deviate from already played games and how strong the players of the played games were, is another idea to give an initial guess of the players strength.
  
  So my rough outline is to create an opening tree to get an initial guess of the strength, ignore the moves played in the opening, weight the accuracy of the moves after the opening phase with the complexity of the position and do some kind of linear regression analysis. Maybe look out not to overvalue time trouble blunders.
  
  I think the key is to get a good "complexity number". The other possibilities you mention … "number of blunders" just reduces "accuracy of play" to something similar with less information. "Depth of tactical traps avoided" is tricky and may computationally be even more expensive.
  
  
  ---
  
  