Answers.Body: <p>Since you've received no replies, I'll attempt an analysis based on first principles. By definition, this has nothing to do with existing work in the field, but it may augment it if any of the ideas presented here are novel.</p>
  <ol>
  <li>The problem definition seems specific, but I judge that it is far too sweeping to be tackled with a single approach. An amalgam of methods seems almost inevitable.</li>
  <li>Engines don't model human thinking. If you want to measure the delta, you need a baseline of what human cognition would produce in order to measure against. For example, which types (FN.1) of moves / positions / themes / priyomes are most elusive for humans?</li>
  </ol>
  <p>FN.1) Obviously, this approach requires a scheme that somehow grades different humans' abilities. An initial model might be categorizing by ELO ranges, but a fixed-range linear breakdown would likely be too granular at low ratings and not granular enough at higher ratings. The performance difference between two players 200 ELO apart at 2500 - 2700 ELO is vastly greater than for two players 200 ELO apart at 1000 - 1200 ELO. The first-principles solution to this is to benchmark problems (that have been precategorized (2)) to measure the solution success rates by ELO range. It's intuitively obvious that some types of problems (say, a moderately difficult combination) that elude all the players in the 1000 - 1200 ELO range would be reliably solved by all the players in the 2500 - 2550 ELO range, say. But the solution to a different type of problem (say, forcing a stalemate using a 6-move sequence) might not even occur to some players in the 2500 - 2550 ELO range.</p>
  <p>FN.2) A number of schemas have been applied to tactics (and even strategic themes), such as &quot;removing the guard&quot;, &quot;trapped piece&quot;, &quot;decoy into fork&quot;, etc. But AFAIK there's no existing knowledge base of the correlation of problems (by this categorization schema) with success rate by player strength. Further, factors such as the differences in the number of moves in a forcing sequence, the availability of refutations (e.g. an Equal-Or-Stronger reply), and the availability of a 'melee' in which the players contest the initiative with competing attacking moves, confound this categorization problem.</p>
  <p>It would seem to make sense to begin with some multi-dimensional schema involving factors like these, and then, through testing vs humans whose ELO is known, determine which factors exhibit influence over the probability of solution success, and rank or scale them for influence. There are statistical methods for exploratory research such as this to economize on the effort, but ultimately a lot of human testing would be needed.</p>
  <p>The only substitute for this would be to use existing game scores, the player's ELO's, and the record of whether the player discovered the solution during the game. The problem is that, if the sample sizes are small, games involve other confounding factors, such as the player's health and mental state, the venue conditions, the stress they're under (e.g. prize determination), proximity of a time control, etc. Controlling for these would likely require (at a guess) using a sample size of 100+ games for each multifactor analysis, each of which had the same factor values (problem type, length of solution, ELO range of the solving player, availability of refutations (3), etc.).</p>
  <p>FN.3) An example of the complexity involved in this problem domain is the phenomenon known as &quot;swindles&quot;. In a swindle, one player plays a suboptimal move that appears to win, and the opponent must find the refutation. Since the swindler is virtually always the higher-rated player, they're engaging in a con. If the opponent finds the correct refutation, they may win some advantage. But the knowledge that the first player is the better player may discourage them from challenging the premise that the move is a good one; it's a psychological ploy. In an identical position where the ELO's are equal (or reversed), the opponent is more likely to try to find the refutation, simply because they're not intimidated. Thus, the <em>relative ELO difference</em> between the players can influence the outcome, not just the ELO range of the player who has to find the solution.</p>
  <p>[First draft, incomplete; will resume with an edit as time permits later]</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-15T19:48:25.113Z
Answers.LastActivityDate: 2024-11-15T19:48:25.113Z
Answers.UserDisplayName: jaxter
Answers.Body: <p>Very interesting question! And I don't think this is a solved problem at the moment.</p>
  <p>One possible approach would be to take a bunch position from historical games, and use an NN to estimate difficulty. Input is position and move, the target is difficulty which is a function of rating of player who made the move and engine eval, e.g, if Carlsen missed that move, it must have high difficulty, if Bob rated 800 found it, it is probably pretty easy.</p>
  <p>But there is a lot of noise in the data, and we might need massive amounts of data for the model to work successfully. And you did mention that you  don't want to rely on &quot;extensive historical data&quot;.</p>
  <p>(I wanted to post this as a comment, but it was too long)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-20T19:20:46.737Z
Answers.LastActivityDate: 2024-11-20T19:20:46.737Z
Answers.UserDisplayName: Akavall
Answers.Body: <p>You could use neural nets (like @Akavall suggested) and try to train on positions to find the probability for every move in a position (Maia Chess already does this basically, you might be able to adapt this if you are able to understand Maia's open source code). A great place to start for training data would be with the Lichess games played database (warning, huge!), but this is definitely not the easiest solution. And I would warn against any approach based on puzzle data, at least if you're using this to analyze games; people think completely differently when they know there's a 'solution' vs. when they're just looking for a good/practical move in a game. But this is just an idea. If you're not that good at programming or even AI programming then this is probably too hard and you should try something else instead.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-21T16:18:01.253Z
Answers.LastActivityDate: 2024-12-21T16:19:15.247Z
Answers.LastEditDate: 2024-12-21T16:19:15.247Z
Answers.UserDisplayName: user43110
Body: <p><strong>TL;DR:</strong> Seeking an automated method to estimate how difficult it is for a human to find specific chess moves suggested by an engine, without human input or extensive historical data. Looking for algorithms, studies, or insights to help develop a reliable difficulty scoring system.</p>
  <hr />
  <p>I'm working on a project that involves real-time analysis of chess games, and I've hit a bit of a roadblock that I'm hoping the community can help me with.</p>
  <p><strong>The Challenge:</strong></p>
  <p>I need to develop a fully automated algorithm that can estimate how difficult it would be for a human player to find a specific move suggested by a chess engine like Stockfish. The goal is to assign a &quot;difficulty score&quot; to the top moves in any given position without any human intervention.</p>
  <p><strong>Constraints:</strong></p>
  <p>Automation: The solution must be entirely automated to handle multiple games simultaneously in real-time.
  No Player Profiles: I can't rely on individual player profiles or extensive historical data. Only minimal historical data like the frequency of a position occurring is acceptable.
  What I've Considered So Far:</p>
  <p><strong>Engine Depth and Time Metrics:</strong></p>
  <p>Using the depth at which the engine finds a move or the time it takes as an indicator of difficulty.
  Evaluation Differences:</p>
  <p>Analyzing the evaluation gap between the best move and other top moves.
  <strong>The Problem:</strong></p>
  <p>I'm struggling to combine these factors into a reliable difficulty score that accurately reflects the human challenge in finding the move. Specifically, I need a way to algorithmically determine when the &quot;best&quot; engine move is not necessarily the most obvious one for a human player.</p>
  <p><strong>Request:</strong></p>
  <p>Algorithms or Methods: Does anyone know of algorithms or computational methods that can estimate move difficulty for humans based on engine analysis?
  Research or References: Are there any studies or papers that delve into quantifying move difficulty in chess?
  Practical Experience: Has anyone tackled a similar problem or have insights from working on chess AI or related projects?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-11-12T15:59:22.303Z
Comments.Text: I'm also interested in this topic, as I am currently working on an engine that would use such features, as the *difficulty*, to choose between moves of the same raw evaluation.
Comments.UserDisplayName: Jean
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-11-16T08:50:12.023Z
Comments.Text: Since you mention depth of calculation: Arguably, width is even worse. (Chess problem solvers routinely solve moremovers with n>10 - if they can be broken down into smaller plans. I know fourmovers that are a horror in comparison, due to the quickly expanding tree of subvariations.)
Comments.UserDisplayName: Hauke Reddmann
Tags: <engines>
Title: Automating the Estimation of Chess Move Difficulty for Human Players
ThreadText: # Automating the Estimation of Chess Move Difficulty for Human Players
  
  (Asked by: username_0 on 2024-11-08)
  
  **TL;DR:** Seeking an automated method to estimate how difficult it is for a human to find specific chess moves suggested by an engine, without human input or extensive historical data. Looking for algorithms, studies, or insights to help develop a reliable difficulty scoring system.
  
  ---
  
  I'm working on a project that involves real-time analysis of chess games, and I've hit a bit of a roadblock that I'm hoping the community can help me with.
  
  **The Challenge:**
  
  I need to develop a fully automated algorithm that can estimate how difficult it would be for a human player to find a specific move suggested by a chess engine like Stockfish. The goal is to assign a "difficulty score" to the top moves in any given position without any human intervention.
  
  **Constraints:**
  
  Automation: The solution must be entirely automated to handle multiple games simultaneously in real-time.
  No Player Profiles: I can't rely on individual player profiles or extensive historical data. Only minimal historical data like the frequency of a position occurring is acceptable.
  What I've Considered So Far:
  
  **Engine Depth and Time Metrics:**
  
  Using the depth at which the engine finds a move or the time it takes as an indicator of difficulty.
  Evaluation Differences:
  
  Analyzing the evaluation gap between the best move and other top moves.
  **The Problem:**
  
  I'm struggling to combine these factors into a reliable difficulty score that accurately reflects the human challenge in finding the move. Specifically, I need a way to algorithmically determine when the "best" engine move is not necessarily the most obvious one for a human player.
  
  **Request:**
  
  Algorithms or Methods: Does anyone know of algorithms or computational methods that can estimate move difficulty for humans based on engine analysis?
  Research or References: Are there any studies or papers that delve into quantifying move difficulty in chess?
  Practical Experience: Has anyone tackled a similar problem or have insights from working on chess AI or related projects?
  
  **Tags:** engines
  
  > **Comments:**
  > **username_1**: I'm also interested in this topic, as I am currently working on an engine that would use such features, as the *difficulty*, to choose between moves of the same raw evaluation.  
  > **username_2**: Since you mention depth of calculation: Arguably, width is even worse. (Chess problem solvers routinely solve moremovers with n>10 - if they can be broken down into smaller plans. I know fourmovers that are a horror in comparison, due to the quickly expanding tree of subvariations.)  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2024-11-15. Score: 1
  
  Since you've received no replies, I'll attempt an analysis based on first principles. By definition, this has nothing to do with existing work in the field, but it may augment it if any of the ideas presented here are novel.
  
  1. The problem definition seems specific, but I judge that it is far too sweeping to be tackled with a single approach. An amalgam of methods seems almost inevitable.
  2. Engines don't model human thinking. If you want to measure the delta, you need a baseline of what human cognition would produce in order to measure against. For example, which types (FN.1) of moves / positions / themes / priyomes are most elusive for humans?
  
  FN.1) Obviously, this approach requires a scheme that somehow grades different humans' abilities. An initial model might be categorizing by ELO ranges, but a fixed-range linear breakdown would likely be too granular at low ratings and not granular enough at higher ratings. The performance difference between two players 200 ELO apart at 2500 - 2700 ELO is vastly greater than for two players 200 ELO apart at 1000 - 1200 ELO. The first-principles solution to this is to benchmark problems (that have been precategorized (2)) to measure the solution success rates by ELO range. It's intuitively obvious that some types of problems (say, a moderately difficult combination) that elude all the players in the 1000 - 1200 ELO range would be reliably solved by all the players in the 2500 - 2550 ELO range, say. But the solution to a different type of problem (say, forcing a stalemate using a 6-move sequence) might not even occur to some players in the 2500 - 2550 ELO range.
  
  FN.2) A number of schemas have been applied to tactics (and even strategic themes), such as "removing the guard", "trapped piece", "decoy into fork", etc. But AFAIK there's no existing knowledge base of the correlation of problems (by this categorization schema) with success rate by player strength. Further, factors such as the differences in the number of moves in a forcing sequence, the availability of refutations (e.g. an Equal-Or-Stronger reply), and the availability of a 'melee' in which the players contest the initiative with competing attacking moves, confound this categorization problem.
  
  It would seem to make sense to begin with some multi-dimensional schema involving factors like these, and then, through testing vs humans whose ELO is known, determine which factors exhibit influence over the probability of solution success, and rank or scale them for influence. There are statistical methods for exploratory research such as this to economize on the effort, but ultimately a lot of human testing would be needed.
  
  The only substitute for this would be to use existing game scores, the player's ELO's, and the record of whether the player discovered the solution during the game. The problem is that, if the sample sizes are small, games involve other confounding factors, such as the player's health and mental state, the venue conditions, the stress they're under (e.g. prize determination), proximity of a time control, etc. Controlling for these would likely require (at a guess) using a sample size of 100+ games for each multifactor analysis, each of which had the same factor values (problem type, length of solution, ELO range of the solving player, availability of refutations (3), etc.).
  
  FN.3) An example of the complexity involved in this problem domain is the phenomenon known as "swindles". In a swindle, one player plays a suboptimal move that appears to win, and the opponent must find the refutation. Since the swindler is virtually always the higher-rated player, they're engaging in a con. If the opponent finds the correct refutation, they may win some advantage. But the knowledge that the first player is the better player may discourage them from challenging the premise that the move is a good one; it's a psychological ploy. In an identical position where the ELO's are equal (or reversed), the opponent is more likely to try to find the refutation, simply because they're not intimidated. Thus, the *relative ELO difference* between the players can influence the outcome, not just the ELO range of the player who has to find the solution.
  
  [First draft, incomplete; will resume with an edit as time permits later]
  
  
  ---
  
  ### Answer by username_4 on 2024-11-20. Score: 0
  
  Very interesting question! And I don't think this is a solved problem at the moment.
  
  One possible approach would be to take a bunch position from historical games, and use an NN to estimate difficulty. Input is position and move, the target is difficulty which is a function of rating of player who made the move and engine eval, e.g, if Carlsen missed that move, it must have high difficulty, if Bob rated 800 found it, it is probably pretty easy.
  
  But there is a lot of noise in the data, and we might need massive amounts of data for the model to work successfully. And you did mention that you don't want to rely on "extensive historical data".
  
  (I wanted to post this as a comment, but it was too long)
  
  
  ---
  
  ### Answer by username_5 on 2024-12-21. Score: 0
  
  You could use neural nets (like @username_4 suggested) and try to train on positions to find the probability for every move in a position (Maia Chess already does this basically, you might be able to adapt this if you are able to understand Maia's open source code). A great place to start for training data would be with the Lichess games played database (warning, huge!), but this is definitely not the easiest solution. And I would warn against any approach based on puzzle data, at least if you're using this to analyze games; people think completely differently when they know there's a 'solution' vs. when they're just looking for a good/practical move in a game. But this is just an idea. If you're not that good at programming or even AI programming then this is probably too hard and you should try something else instead.
  
  
  ---
  
  