Answers.Body: <p>I verified your results through game matches.</p>
  <h3>d3 vs d1</h3>
  <p>Here d1 wins. I am surprise with this result.</p>
  <pre><code>Score of sf14.1_d3 vs sf14.1_d1: 423 - 463 - 114  [0.480] 1000
  ...      sf14.1_d3 playing White: 245 - 196 - 59  [0.549] 500
  ...      sf14.1_d3 playing Black: 178 - 267 - 55  [0.411] 500
  ...      White vs Black: 512 - 374 - 114  [0.569] 1000
  Elo difference: -13.9 +/- 20.3, LOS: 9.0 %, DrawRatio: 11.4 %
  </code></pre>
  <h3>d4 vs d1</h3>
  <p>d4 now wins.</p>
  <pre><code>Score of sf14.1_d4 vs sf14.1_d1: 665 - 237 - 98  [0.714] 1000
  ...      sf14.1_d4 playing White: 336 - 113 - 51  [0.723] 500
  ...      sf14.1_d4 playing Black: 329 - 124 - 47  [0.705] 500
  ...      White vs Black: 460 - 442 - 98  [0.509] 1000
  Elo difference: 158.9 +/- 22.4, LOS: 100.0 %, DrawRatio: 9.8 %
  </code></pre>
  <h3>Use sf14</h3>
  <p>d3 wins</p>
  <pre><code>Score of sf14_d3 vs sf14_d1: 564 - 337 - 99  [0.614] 1000
  ...      sf14_d3 playing White: 296 - 156 - 48  [0.640] 500
  ...      sf14_d3 playing Black: 268 - 181 - 51  [0.587] 500
  ...      White vs Black: 477 - 424 - 99  [0.526] 1000
  Elo difference: 80.3 +/- 20.9, LOS: 100.0 %, DrawRatio: 9.9 %
  </code></pre>
  <p>So this has to do with the changes in Stockfish itself.</p>
  <p>I checked the code compiling and testing every commit starting from the release of sf14, that has to do with pruning/reduction. The commit that is responsible on the d3 loss vs d1 is this:
  <a href="https://github.com/official-stockfish/Stockfish/commit/ad357e147a1b8481a04761d726ce1db14115a68f" rel="nofollow noreferrer">https://github.com/official-stockfish/Stockfish/commit/ad357e147a1b8481a04761d726ce1db14115a68f</a></p>
  <h3>d3 vs d1</h3>
  <p>The 20210828 in sf20210828_d3 is the commit date .</p>
  <pre><code>Score of sf20210828_d3 vs sf20210828_d1: 393 - 478 - 129  [0.458] 1000
  ...      sf20210828_d3 playing White: 216 - 219 - 65  [0.497] 500
  ...      sf20210828_d3 playing Black: 177 - 259 - 64  [0.418] 500
  ...      White vs Black: 475 - 396 - 129  [0.539] 1000
  Elo difference: -29.6 +/- 20.2, LOS: 0.2 %, DrawRatio: 12.9 %
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-08T19:43:07.847Z
Answers.Text: Thank you! Nice to know that I didn't just make an error. I still would be curious to know what changes lead to this happening.
Answers.UserDisplayName: Evan Russek
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-09T00:44:17.610Z
Answers.Text: @EvanRussek updated the answer showing which commit defeats the d3 to d1.
Answers.UserDisplayName: ferdy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-09T13:18:18.503Z
Answers.Text: (+1) digging to the commit level details.
Answers.UserDisplayName: SecretAgentMan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-09T18:22:26.617Z
Answers.Text: @ferdy thank you!
Answers.UserDisplayName: Evan Russek
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-08T17:13:07.793Z
Answers.LastActivityDate: 2022-02-09T00:42:55.657Z
Answers.LastEditDate: 2022-02-09T00:42:55.657Z
Answers.UserDisplayName: ferdy
Body: <p>I have been working on a project looking at how stockfish moves change with increasing computation time (here search depth), and have found something peculiar - namely that the quality of moves that stockfish selects does not increase monotonically with depth.</p>
  <p>The question is - why are the moves that stockfish selects at depth 3 worse than the moves that stockfish selects at depth 1?</p>
  <p>A bit on how i’ve analyzed this. I’ve taken many positions for games appearing online, and then for each position collected stockfish’s move suggestions in that position, at each sequentially increasing depth limit (so getting the move suggested at depth 1, depth 2, depth 3 etc... up to depth 18). Then I have evaluated each of those suggested moves by carrying out the move and using stockfish (at depth limit 18) to evaluate the resultant board state.</p>
  <p>The plot below shows the average evaluations of the move stockfish selects at each increasing depth. (Note that for each position, I translated the evaluation to win probability, and then subtracted the evaluation of the move at depth 1 from the rest of the line so it would be at 0. Additionally, I have cleared the cache before each evaluation to try to remove any ordering effects).</p>
  <p><a href="https://i.sstatic.net/veO0c.png" rel="noreferrer"><img src="https://i.sstatic.net/veO0c.png" alt="enter image description here" /></a></p>
  <p>Curiously, the evaluations are non-monotonic with respect to depth. That is, according to the evaluations of stockfish at depth 18, the moves selected by stockfish at depth 3 are worse than the moves selected at depth 1.</p>
  <p>Would anyone understand why this happens? I would have expected, on average, move quality to increase with increasing search depth.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-08T05:54:56.623Z
Comments.Text: Depth = 18 is hardly the truth about a position.  What is your nodes/sec ?
Comments.UserDisplayName: SecretAgentMan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-08T06:29:35.873Z
Comments.Text: Which stockfish version did you use?
Comments.UserDisplayName: ferdy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-08T07:33:03.747Z
Comments.Text: why would you expect on average for a move quality to increase with increasing search depth?
Comments.UserDisplayName: Anton Menshov
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-08T11:27:49.600Z
Comments.Text: Do you impose a time limit? (A user interface to Stockfish might impose one, with a default value which is low enough that it would be typical for Stockfish's search to be incomplete.)
Comments.UserDisplayName: Rosie F
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-08T16:42:37.163Z
Comments.Text: @ferdy I used the newest version - 14.1
Comments.UserDisplayName: Evan Russek
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-09T00:11:05.727Z
Comments.Text: @AntonMenshov increasing search depth is effectively the same as increased thinking time. It should be intuitively obvious that engines (or humans for that matter) play better with more time.
Comments.UserDisplayName: Allure
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-09T00:11:33.727Z
Comments.Text: @RosieF since the search is to fixed depth there is no time limit.
Comments.UserDisplayName: Allure
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-09T05:56:31.813Z
Comments.Text: Related: [Bias-Variance tradeoff](https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff).
Comments.UserDisplayName: Mateen Ulhaq
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-09T13:40:08.970Z
Comments.Text: Have you tried playing a match between SF at 1ply Vs SF at 3ply? I guess thousands of games could be played quite fast.
Comments.UserDisplayName: emdio
Tags: <stockfish><evaluation><depth>
Title: Why does stockfish move quality increase non-monotonically with increasing depth? -- selected moves at depth 3 are worse than depth 1
ThreadText: # Why does stockfish move quality increase non-monotonically with increasing depth? -- selected moves at depth 3 are worse than depth 1
  
  (Asked by: username_0 on 2022-02-08)
  
  I have been working on a project looking at how stockfish moves change with increasing computation time (here search depth), and have found something peculiar - namely that the quality of moves that stockfish selects does not increase monotonically with depth.
  
  The question is - why are the moves that stockfish selects at depth 3 worse than the moves that stockfish selects at depth 1?
  
  A bit on how i’ve analyzed this. I’ve taken many positions for games appearing online, and then for each position collected stockfish’s move suggestions in that position, at each sequentially increasing depth limit (so getting the move suggested at depth 1, depth 2, depth 3 etc... up to depth 18). Then I have evaluated each of those suggested moves by carrying out the move and using stockfish (at depth limit 18) to evaluate the resultant board state.
  
  The plot below shows the average evaluations of the move stockfish selects at each increasing depth. (Note that for each position, I translated the evaluation to win probability, and then subtracted the evaluation of the move at depth 1 from the rest of the line so it would be at 0. Additionally, I have cleared the cache before each evaluation to try to remove any ordering effects).
  
  [![enter image description here](https://i.sstatic.net/veO0c.png)](https://i.sstatic.net/veO0c.png)
  
  Curiously, the evaluations are non-monotonic with respect to depth. That is, according to the evaluations of stockfish at depth 18, the moves selected by stockfish at depth 3 are worse than the moves selected at depth 1.
  
  Would anyone understand why this happens? I would have expected, on average, move quality to increase with increasing search depth.
  
  **Tags:** stockfish, evaluation, depth
  
  > **Comments:**
  > **username_1**: Depth = 18 is hardly the truth about a position.  What is your nodes/sec ?  
  > **username_2**: Which stockfish version did you use?  
  > **username_3**: why would you expect on average for a move quality to increase with increasing search depth?  
  > **username_4**: Do you impose a time limit? (A user interface to Stockfish might impose one, with a default value which is low enough that it would be typical for Stockfish's search to be incomplete.)  
  > **username_0**: @username_2 I used the newest version - 14.1  
  > **username_5**: @AntonMenshov increasing search depth is effectively the same as increased thinking time. It should be intuitively obvious that engines (or humans for that matter) play better with more time.  
  > **username_5**: @RosieF since the search is to fixed depth there is no time limit.  
  > **username_6**: Related: [Bias-Variance tradeoff](https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff).  
  > **username_7**: Have you tried playing a match between SF at 1ply Vs SF at 3ply? I guess thousands of games could be played quite fast.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-02-08. Score: 6
  
  I verified your results through game matches.
  d3 vs d1
  Here d1 wins. I am surprise with this result.
  Score of sf14.1_d3 vs sf14.1_d1: 423 - 463 - 114  [0.480] 1000
  ...      sf14.1_d3 playing White: 245 - 196 - 59  [0.549] 500
  ...      sf14.1_d3 playing Black: 178 - 267 - 55  [0.411] 500
  ...      White vs Black: 512 - 374 - 114  [0.569] 1000
  Elo difference: -13.9 +/- 20.3, LOS: 9.0 %, DrawRatio: 11.4 %
  
  d4 vs d1
  d4 now wins.
  Score of sf14.1_d4 vs sf14.1_d1: 665 - 237 - 98  [0.714] 1000
  ...      sf14.1_d4 playing White: 336 - 113 - 51  [0.723] 500
  ...      sf14.1_d4 playing Black: 329 - 124 - 47  [0.705] 500
  ...      White vs Black: 460 - 442 - 98  [0.509] 1000
  Elo difference: 158.9 +/- 22.4, LOS: 100.0 %, DrawRatio: 9.8 %
  
  Use sf14
  d3 wins
  Score of sf14_d3 vs sf14_d1: 564 - 337 - 99  [0.614] 1000
  ...      sf14_d3 playing White: 296 - 156 - 48  [0.640] 500
  ...      sf14_d3 playing Black: 268 - 181 - 51  [0.587] 500
  ...      White vs Black: 477 - 424 - 99  [0.526] 1000
  Elo difference: 80.3 +/- 20.9, LOS: 100.0 %, DrawRatio: 9.9 %
  
  So this has to do with the changes in Stockfish itself.
  I checked the code compiling and testing every commit starting from the release of sf14, that has to do with pruning/reduction. The commit that is responsible on the d3 loss vs d1 is this:
  https://github.com/official-stockfish/Stockfish/commit/ad357e147a1b8481a04761d726ce1db14115a68f
  d3 vs d1
  The 20210828 in sf20210828_d3 is the commit date .
  Score of sf20210828_d3 vs sf20210828_d1: 393 - 478 - 129  [0.458] 1000
  ...      sf20210828_d3 playing White: 216 - 219 - 65  [0.497] 500
  ...      sf20210828_d3 playing Black: 177 - 259 - 64  [0.418] 500
  ...      White vs Black: 475 - 396 - 129  [0.539] 1000
  Elo difference: -29.6 +/- 20.2, LOS: 0.2 %, DrawRatio: 12.9 %
  
  > **Comments:**
  > **username_0**: Thank you! Nice to know that I didn't just make an error. I still would be curious to know what changes lead to this happening.  
  > **username_2**: @EvanRussek updated the answer showing which commit defeats the d3 to d1.  
  > **username_1**: (+1) digging to the commit level details.  
  > **username_0**: @username_2 thank you!  
  
  ---
  
  