Answers.Body: <p>Different engines have different "scales" for their numerical evaluations. For instance, in a typical middlegame position with plenty of play left, when Houdini says +2.00 or better, it is highly probable that White has a winning advantage (though even here I've included qualifications for a reason). But consider: one could modify the source code of Houdini and double the absolute values of all numbers involved in evaluations; one gets an engine of identical strength that produces identical play, but now +4.00 means what +2.00 used to mean. This illustrates that one shouldn't expect a uniform numerical threshold <em>across engines</em> that typically indicates a winning advantage.</p>
  
  <p>More than this, though, it's important to understand that a numerical engine evaluation of a position (as opposed to an outright declaration of inevitable mate) <em>never</em> strictly translates to "a won game," even for a single, fixed engine. A key point is that numerical evaluations have no clear-cut "meaning" in broad chess terms, and are rather just a substitute for sentient thought that is used to mechanically guide an engine toward generally desirable outcomes by influencing which move it selects at each point in the game; in this light, what is ultimately most important to an engine's play is just the <em>difference</em> in evaluation assigned to potential moves, rather than anything about the <em>absolute values</em> involved. The numbers are useful to the engine itself, which needs something that concrete in order to make a decision for one move over another, but we humans shouldn't be too quick to read more meaning into the magnitudes involved with thoughts like "+X means a win."</p>
  
  <p>In particular, the further and further we get toward an endgame as opposed to a middlegame, the less we can use a rule of thumb (like my +2.00 for Houdini in middlegames above) about a certain threshold being enough for a win. One key reason for this is the difficulty that engines have recognizing fortresses, where an abundance of extra material still isn't enough to win. For instance, when I feed Stockfish this position,</p>
  
  <pre><code>[FEN "5k2/6p1/5r2/3Q3P/4K3/8/8/8 w - - 0 1"]
  </code></pre>
  
  <p>after a couple minutes thought it is giving an evaluation of about +7.00, and in a typical position, when Stockfish says that, you almost certainly do have a win on your hands. Nevertheless, this is a dead draw, and a human can see this easily once the fact is realized that Black can just shuffle the rook between f6 and h6, and so (1) the h-pawn is useless, and (2) the white king will never be able to help the white queen attack. <em>Eventually</em>, Stockfish will recognize a draw here too, once it butts up against 50 moves, say, or finally runs out of different moves to try and finally can't avoid a repetition, but those events are way down the search depth line.</p>
  
  <p>The endgame position from your earlier question that you linked to is akin to this sort of fortress, in that the extra connected passed pawns White has there are nice and all, but ultimately not quite enough to win in that position. If an engine were to calculate for enough time to see as much information as is contained in tablebases, then its evaluation would come down to 0, but in the meantime, its evaluation algorithm has nothing better to go on than to give a + for that extra material (that it doesn't yet know is meaningless).</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-01-01T18:57:49.373Z
Answers.Text: +1  for *"More than this, though, it's important to understand that a numerical engine evaluation of a position (as opposed to an outright declaration of inevitable mate) never strictly translates to a won game"*
Answers.UserDisplayName: ferit
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2012-12-30T19:42:18.797Z
Answers.LastActivityDate: 2019-08-19T16:54:05.183Z
Answers.LastEditDate: 2019-08-19T16:54:05.183Z
Answers.UserDisplayName: ETD
Answers.Body: <p>I think this picture describes the situation quite well. It was created from 400k games, and considers only plain piece material.</p>
  
  <p><img src="https://i.sstatic.net/fNxTD.png" alt="Win Probability/Pawn Advantage"></p>
  
  <p>Source: <a href="https://www.chessprogramming.org/Pawn_Advantage,_Win_Percentage,_and_Elo" rel="nofollow noreferrer" title="Pawn Advantage, Win Percentage, and ELO">Pawn Advantage, Win Percentage, and ELO</a></p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-01-01T19:01:27.960Z
Answers.Text: Nice contribution! +1
Answers.UserDisplayName: ferit
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-27T19:33:49.227Z
Answers.Text: @Thomas Ahle: The graph is interesting. But the original article is no longer available, the wikispaces link has sadly gone down. Do you remember the exact meaning of W=Win Probability? Was it win vs lose ignoring the draws? Or was it "expected score" taking the draws into account?
Answers.UserDisplayName: Diedrsch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-23T14:49:54.357Z
Answers.Text: @Diedrsch I've updated the link
Answers.UserDisplayName: Thomas Ahle
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2014-03-09T18:04:32.280Z
Answers.LastActivityDate: 2019-08-23T14:49:44.490Z
Answers.LastEditDate: 2019-08-23T14:49:44.490Z
Answers.UserDisplayName: Thomas Ahle
Answers.Body: <p>There are a couple of things here.</p>
  
  <p>First, each program is going to have its own way of evaluating positions so the scores can't be directly compared.  For example, I was running StockFish against Rybka recently and found Stockfish's scores were about double those of Rybka.  I was surprised by this, but it's pretty clear that a score of 1 doesn't always mean "1 pawn."  I think what we should look at is how the score changes.  Another curiosity I saw yesterday (while answering one of your other questions, coincidentally) was that Stockfish's evaluation algorithm doesn't like odd numbers very much.  In fact most scores were multiples of .04.  Given that the magnitude of the value is arbitrary, I would not assume any certain value means "a sure win" unless the machine is claiming it found a mate.</p>
  
  <p>Second, end game tablebases were created because solving end games requires a lot of search depth.  Computers playing at tournament speeds just don't do it well.  I was working through a different game a few days back and announced on this site that one side had an advantage.  Ed used a tablebase to show there was no mystery left in the position - it was theoretically drawn.  Of course, there's a huge difference between drawn-with-perfect-play and drawn; the players have to find the right moves.</p>
  
  <hr>
  
  <p>The small value generally given to White in the early stages of the game basically means white can claim more valuable real estate.  For example, on move 1, White can claim e4 and attack e5 and f5.  Black can counter. But then White can play Nc3 and attack/bolster a4, b5, d5, and e4.  But Black can counter. So it means very little.</p>
  
  <hr>
  
  <p>Finally, to answer the question in your subject line - the evaluations are very trustworthy as they are based on hard facts and an impressive search depth.  Of course, the machines aren't infallible. But we b-players must remember that Stockfish (or Rybka) play at GM strengths on modest hardware.  On the best <em>common</em> hardware, they estimate their ratings at FIDE 3200. This is so high that only the best humans have a slight chance of not losing.</p>
  
  <p>Consider what this means; I (USCF 1650-ish) have <em>no</em> chance against a person (say, USCF 2050) who has <em>no</em> chance against a person (say, USCF 2450) who has a <em>no</em> chance against a person (say, USCF 2850) who has a <em>sliver of a</em> chance against a top-flight commercial program (FIDE 3200).</p>
  
  <p>Thus, when Stockfish says one move is better than another, I usually take it at face value.   When I hook up the endgame tablebases this thing's gonna start announcing mate-in-30s, lol.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-01-01T19:00:44.463Z
Answers.Text: +1 for *"I would not assume any certain value means "a sure win" unless the machine is claiming it found a mate."*
Answers.UserDisplayName: ferit
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2012-12-30T21:38:05.380Z
Answers.Text: Very nice response. I always thought that an evaluation of 1 meant 1 pawn's worth of material. Also, chesstempo says that the best move(s) in its problems are ones that win at least 2 pawns worth of material, so I considered an engine evaluation of +2 or over to be winning regardless of the stage in a game. However, I have found the analysis of stockfish to be faulty before and seen how it fails to assess endgames properly. On that note, do you know where I can find a tablebase of endgames?
Answers.UserDisplayName: chubbycantorset
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2012-12-30T22:54:01.120Z
Answers.Text: Here's the online 6-man tablebase Ed posted: http://www.k4it.de/index.php?topic=egtb&lang=en
Answers.UserDisplayName: Tony Ennis
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2012-12-30T15:06:34.000Z
Answers.LastActivityDate: 2012-12-30T18:09:18.277Z
Answers.LastEditDate: 2012-12-30T18:09:18.277Z
Answers.UserDisplayName: Tony Ennis
Body: <p>Fritz 12 with Rybka gave a friend of mine an evaluation of +3 for white in <a href="https://chess.stackexchange.com/questions/1707/is-this-endgame-position-a-draw">this endgame position</a>, </p>
  
  <pre><code>[Title "White to move"]
  [FEN "8/8/8/3k3p/3P3K/2P3R1/8/7r w - - 0 1"]
  </code></pre>
  
  <p>which turned out to be a draw. But I've heard that +3 from a computer means a guaranteed win with perfect play. Did I just hear wrong? How should computer evaluations be interpreted in general? What do opening advantages of &lt;.5 even mean?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-29T12:35:22.647Z
Comments.Text: For the record, I checked this with the Lichess 7-piece tablebases, and it is, indeed, a draw.
Comments.UserDisplayName: PhishMaster
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2012-12-31T06:31:55.453Z
Comments.Text: I would challenge the "guaranteed win" comment.  The evaluation number is a heuristic indicator, basically a "feeling" that the computer has for the position.  Endgames can have "surprising" outcomes, and unless the computer has been programmed to detect all possible patterns (or it can calculate all the way), it will miss some.  To look at it another way, if +3 was really guaranteed, it wouldn't be +3, it'd be +infinity.
Comments.UserDisplayName: Daniel B
Tags: <analysis><engines><software><statistics><evaluation>
Title: Computer evaluations: How trustworthy are they?
ThreadText: # Computer evaluations: How trustworthy are they?
  
  (Asked by: username_0 on 2012-12-30)
  
  Fritz 12 with Rybka gave a friend of mine an evaluation of +3 for white in [this endgame position](https://chess.stackexchange.com/questions/1707/is-this-endgame-position-a-draw),
  
  ```
  [Title "White to move"]
  [FEN "8/8/8/3k3p/3P3K/2P3R1/8/7r w - - 0 1"]
  
  ```
  
  which turned out to be a draw. But I've heard that +3 from a computer means a guaranteed win with perfect play. Did I just hear wrong? How should computer evaluations be interpreted in general? What do opening advantages of <.5 even mean?
  
  **Tags:** analysis, engines, software, statistics, evaluation
  
  > **Comments:**
  > **username_2**: I would challenge the "guaranteed win" comment.  The evaluation number is a heuristic indicator, basically a "feeling" that the computer has for the position.  Endgames can have "surprising" outcomes, and unless the computer has been programmed to detect all possible patterns (or it can calculate all the way), it will miss some.  To look at it another way, if +3 was really guaranteed, it wouldn't be +3, it'd be +infinity.  
  > **username_1**: For the record, I checked this with the Lichess 7-piece tablebases, and it is, indeed, a draw.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_7 on 2012-12-30. Score: 8 | Accepted answer
  
  There are a couple of things here.
  
  First, each program is going to have its own way of evaluating positions so the scores can't be directly compared. For example, I was running StockFish against Rybka recently and found Stockfish's scores were about double those of Rybka. I was surprised by this, but it's pretty clear that a score of 1 doesn't always mean "1 pawn." I think what we should look at is how the score changes. Another curiosity I saw yesterday (while answering one of your other questions, coincidentally) was that Stockfish's evaluation algorithm doesn't like odd numbers very much. In fact most scores were multiples of .04. Given that the magnitude of the value is arbitrary, I would not assume any certain value means "a sure win" unless the machine is claiming it found a mate.
  
  Second, end game tablebases were created because solving end games requires a lot of search depth. Computers playing at tournament speeds just don't do it well. I was working through a different game a few days back and announced on this site that one side had an advantage. Ed used a tablebase to show there was no mystery left in the position - it was theoretically drawn. Of course, there's a huge difference between drawn-with-perfect-play and drawn; the players have to find the right moves.
  
  ---
  
  The small value generally given to White in the early stages of the game basically means white can claim more valuable real estate. For example, on move 1, White can claim e4 and attack e5 and f5. Black can counter. But then White can play Nc3 and attack/bolster a4, b5, d5, and e4. But Black can counter. So it means very little.
  
  ---
  
  Finally, to answer the question in your subject line - the evaluations are very trustworthy as they are based on hard facts and an impressive search depth. Of course, the machines aren't infallible. But we b-players must remember that Stockfish (or Rybka) play at GM strengths on modest hardware. On the best *common* hardware, they estimate their ratings at FIDE 3200. This is so high that only the best humans have a slight chance of not losing.
  
  Consider what this means; I (USCF 1650-ish) have *no* chance against a person (say, USCF 2050) who has *no* chance against a person (say, USCF 2450) who has a *no* chance against a person (say, USCF 2850) who has a *sliver of a* chance against a top-flight commercial program (FIDE 3200).
  
  Thus, when Stockfish says one move is better than another, I usually take it at face value. When I hook up the endgame tablebases this thing's gonna start announcing mate-in-30s, lol.
  
  > **Comments:**
  > **username_0**: Very nice response. I always thought that an evaluation of 1 meant 1 pawn's worth of material. Also, chesstempo says that the best move(s) in its problems are ones that win at least 2 pawns worth of material, so I considered an engine evaluation of +2 or over to be winning regardless of the stage in a game. However, I have found the analysis of stockfish to be faulty before and seen how it fails to assess endgames properly. On that note, do you know where I can find a tablebase of endgames?  
  > **username_7**: Here's the online 6-man tablebase Ed posted: http://www.k4it.de/index.php?topic=egtb&lang=en  
  > **username_4**: +1 for *"I would not assume any certain value means "a sure win" unless the machine is claiming it found a mate."*  
  
  ---
  
  ### Answer by username_3 on 2012-12-30. Score: 15
  
  Different engines have different "scales" for their numerical evaluations. For instance, in a typical middlegame position with plenty of play left, when Houdini says +2.00 or better, it is highly probable that White has a winning advantage (though even here I've included qualifications for a reason). But consider: one could modify the source code of Houdini and double the absolute values of all numbers involved in evaluations; one gets an engine of identical strength that produces identical play, but now +4.00 means what +2.00 used to mean. This illustrates that one shouldn't expect a uniform numerical threshold *across engines* that typically indicates a winning advantage.
  
  More than this, though, it's important to understand that a numerical engine evaluation of a position (as opposed to an outright declaration of inevitable mate) *never* strictly translates to "a won game," even for a single, fixed engine. A key point is that numerical evaluations have no clear-cut "meaning" in broad chess terms, and are rather just a substitute for sentient thought that is used to mechanically guide an engine toward generally desirable outcomes by influencing which move it selects at each point in the game; in this light, what is ultimately most important to an engine's play is just the *difference* in evaluation assigned to potential moves, rather than anything about the *absolute values* involved. The numbers are useful to the engine itself, which needs something that concrete in order to make a decision for one move over another, but we humans shouldn't be too quick to read more meaning into the magnitudes involved with thoughts like "+X means a win."
  
  In particular, the further and further we get toward an endgame as opposed to a middlegame, the less we can use a rule of thumb (like my +2.00 for Houdini in middlegames above) about a certain threshold being enough for a win. One key reason for this is the difficulty that engines have recognizing fortresses, where an abundance of extra material still isn't enough to win. For instance, when I feed Stockfish this position,
  
  ```
  [FEN "5k2/6p1/5r2/3Q3P/4K3/8/8/8 w - - 0 1"]
  
  ```
  
  after a couple minutes thought it is giving an evaluation of about +7.00, and in a typical position, when Stockfish says that, you almost certainly do have a win on your hands. Nevertheless, this is a dead draw, and a human can see this easily once the fact is realized that Black can just shuffle the rook between f6 and h6, and so (1) the h-pawn is useless, and (2) the white king will never be able to help the white queen attack. *Eventually*, Stockfish will recognize a draw here too, once it butts up against 50 moves, say, or finally runs out of different moves to try and finally can't avoid a repetition, but those events are way down the search depth line.
  
  The endgame position from your earlier question that you linked to is akin to this sort of fortress, in that the extra connected passed pawns White has there are nice and all, but ultimately not quite enough to win in that position. If an engine were to calculate for enough time to see as much information as is contained in tablebases, then its evaluation would come down to 0, but in the meantime, its evaluation algorithm has nothing better to go on than to give a + for that extra material (that it doesn't yet know is meaningless).
  
  > **Comments:**
  > **username_4**: +1  for *"More than this, though, it's important to understand that a numerical engine evaluation of a position (as opposed to an outright declaration of inevitable mate) never strictly translates to a won game"*  
  
  ---
  
  ### Answer by username_5 on 2014-03-09. Score: 10
  
  I think this picture describes the situation quite well. It was created from 400k games, and considers only plain piece material.
  
  ![Win Probability/Pawn Advantage](https://i.sstatic.net/fNxTD.png)
  
  Source: [Pawn Advantage, Win Percentage, and ELO](https://www.chessprogramming.org/Pawn_Advantage,_Win_Percentage,_and_Elo "Pawn Advantage, Win Percentage, and ELO")
  
  > **Comments:**
  > **username_4**: Nice contribution! +1  
  > **username_6**: @username_5: The graph is interesting. But the original article is no longer available, the wikispaces link has sadly gone down. Do you remember the exact meaning of W=Win Probability? Was it win vs lose ignoring the draws? Or was it "expected score" taking the draws into account?  
  > **username_5**: @username_6 I've updated the link  
  
  ---
  
  