Answers.Body: <p>If Stockfish was a pure minimax/alpha-beta searcher with all leaves having a predetermined depth, your assumption that the search result should be reproducible would hold true, even with a different state of the hash table. That is because the result in minimax/alpha-beta only depends on the evaluation function, but not on the order in which the moves are searched, which only has an impact on speed.</p>
  
  <p>However, in fact Stockfish is using a lot of <a href="https://www.chessprogramming.org/Selectivity" rel="noreferrer">forward pruning, reduction and extension techniques</a> that are state/history dependent, so the results of the search of a subtree depends on the state/history with which it entered the subtree. Therefore, starting with hash table entries that influence move ordering (and can actually return a different evaluation than searching the subtree again) leads to a different result of for a search of same depth.</p>
  
  <p>There is one more detail in the Stockfish case why even with pure alpha-beta search re-searching the position again would give different results. The <a href="https://github.com/official-stockfish/Stockfish/blob/8fec8834715a440ac18e24e130888c2c60bab352/src/search.cpp#L88-L92" rel="noreferrer">evaluation of draws is pseudo-randomized</a> with small fluctuation from zero in order to avoid some search artefacts in case of repetition draws, and this pseudo-random number depends on the node count of the current search. Therefore, using a different move ordering due to the hash entries would lead to different pseudo-random numbers, and hence potentially a different search result.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-01T09:16:00.923Z
Answers.Text: Very interesting, I wasn't aware of forward pruning. I know alpha-beta engines do move ordering (the ability to do this is the main advantage of a transposition table). But what you're saying is that the subtrees of the lowest ranked moves are ignored altogether, and not just looked at last?
Answers.UserDisplayName: Inertial Ignorance
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-01T09:21:58.323Z
Answers.Text: Yes, this is why modern chess engines can sometimes be blind to solutions of quite artificial studies, because they might just prune the best move (e.g., because it is sacrificing a piece for seemingly nothing). However, these pruning techniques only apply to small remaining depths, so as the total depth grows, it gets more and more unlikely that the critical move is pruned, so the search should still converge to the correct move.
Answers.UserDisplayName: Fabian Fichter
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-01T09:31:01.590Z
Answers.Text: But the details of the pruning, reduction and extension techniques would probably go beyond the scope of this thread. https://www.chessprogramming.org/Selectivity should be a starting point, otherwise you can maybe ask it as a new question.
Answers.UserDisplayName: Fabian Fichter
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-01T19:36:55.723Z
Answers.Text: Makes sense. I figured the pruning would only happen when the engine is near the depth limit, since if a move doesn't have a high evaluation at that point it's most likely bad.
Answers.UserDisplayName: Inertial Ignorance
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-01T09:01:27.960Z
Answers.LastActivityDate: 2019-09-01T09:42:10.700Z
Answers.LastEditDate: 2019-09-01T09:42:10.700Z
Answers.UserDisplayName: Fabian Fichter
Body: <p>When I let Stockfish get to a certain depth (say depth 33), it gives some evaluation. Then I refresh the position (while leaving Stockfish on) and when it gets to the same depth it gives a different evaluation.</p>
  
  <p>I know it uses previous analysis stored in a hash table to get to the same depth quicker the second time around, but why it can give a different evaluation puzzles me. Previous analysis should only improve search speed for a deterministic engine.</p>
  
  <p>For anyone familiar with Stockfish's source code, any reasons this could be?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-01T08:18:18.217Z
Comments.Text: The answers in [randomness in engine play](https://chess.stackexchange.com/questions/19482/randomness-in-engine-play) aren't clear as why when reaching the same search depth, in two or more analysis of the same position, Stockfish give different evaluations. 
  
  Even more, at a greater depth of analysis the invariant movements between the calculated variants are less and less, until reaching a minimum of 31 plays deep.
Comments.UserDisplayName: djnavas
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-01T09:09:00.040Z
Comments.Text: This question is not a duplicate. It is assuming a deterministic search without timing and multi-threading related non-determinism, so the answers from the other thread do not apply here.
Comments.UserDisplayName: Fabian Fichter
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-02T07:51:53.183Z
Comments.Text: Short answer: Stockfish is not God
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-02T08:42:23.763Z
Comments.Text: @David An entity giving consistent evaluations doesn't imply it's god :)
Comments.UserDisplayName: Inertial Ignorance
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-11T20:53:02.410Z
Comments.Text: it also depends on how you run stockfish, if you stop it manually by clicking on a graphical interface then you risk not stopping at exactly the same point in the eval.
Comments.UserDisplayName: Purefan
Tags: <engines><analysis><stockfish>
Title: Why can Stockfish give different evaluations at the same depth?
ThreadText: # Why can Stockfish give different evaluations at the same depth?
  
  (Asked by: username_0 on 2019-09-01)
  
  When I let Stockfish get to a certain depth (say depth 33), it gives some evaluation. Then I refresh the position (while leaving Stockfish on) and when it gets to the same depth it gives a different evaluation.
  
  I know it uses previous analysis stored in a hash table to get to the same depth quicker the second time around, but why it can give a different evaluation puzzles me. Previous analysis should only improve search speed for a deterministic engine.
  
  For anyone familiar with Stockfish's source code, any reasons this could be?
  
  **Tags:** engines, analysis, stockfish
  
  > **Comments:**
  > **username_1**: The answers in [randomness in engine play](https://chess.stackexchange.com/questions/19482/randomness-in-engine-play) aren't clear as why when reaching the same search depth, in two or more analysis of the same position, Stockfish give different evaluations.   Even more, at a greater depth of analysis the invariant movements between the calculated variants are less and less, until reaching a minimum of 31 plays deep.  
  > **username_2**: This question is not a duplicate. It is assuming a deterministic search without timing and multi-threading related non-determinism, so the answers from the other thread do not apply here.  
  > **username_3**: Short answer: Stockfish is not God  
  > **username_0**: @username_3 An entity giving consistent evaluations doesn't imply it's god :)  
  > **username_4**: it also depends on how you run stockfish, if you stop it manually by clicking on a graphical interface then you risk not stopping at exactly the same point in the eval.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-09-01. Score: 7 | Accepted answer
  
  If Stockfish was a pure minimax/alpha-beta searcher with all leaves having a predetermined depth, your assumption that the search result should be reproducible would hold true, even with a different state of the hash table. That is because the result in minimax/alpha-beta only depends on the evaluation function, but not on the order in which the moves are searched, which only has an impact on speed.
  
  However, in fact Stockfish is using a lot of [forward pruning, reduction and extension techniques](https://www.chessprogramming.org/Selectivity) that are state/history dependent, so the results of the search of a subtree depends on the state/history with which it entered the subtree. Therefore, starting with hash table entries that influence move ordering (and can actually return a different evaluation than searching the subtree again) leads to a different result of for a search of same depth.
  
  There is one more detail in the Stockfish case why even with pure alpha-beta search re-searching the position again would give different results. The [evaluation of draws is pseudo-randomized](https://github.com/official-stockfish/Stockfish/blob/8fec8834715a440ac18e24e130888c2c60bab352/src/search.cpp#L88-L92) with small fluctuation from zero in order to avoid some search artefacts in case of repetition draws, and this pseudo-random number depends on the node count of the current search. Therefore, using a different move ordering due to the hash entries would lead to different pseudo-random numbers, and hence potentially a different search result.
  
  > **Comments:**
  > **username_0**: Very interesting, I wasn't aware of forward pruning. I know alpha-beta engines do move ordering (the ability to do this is the main advantage of a transposition table). But what you're saying is that the subtrees of the lowest ranked moves are ignored altogether, and not just looked at last?  
  > **username_2**: Yes, this is why modern chess engines can sometimes be blind to solutions of quite artificial studies, because they might just prune the best move (e.g., because it is sacrificing a piece for seemingly nothing). However, these pruning techniques only apply to small remaining depths, so as the total depth grows, it gets more and more unlikely that the critical move is pruned, so the search should still converge to the correct move.  
  > **username_2**: But the details of the pruning, reduction and extension techniques would probably go beyond the scope of this thread. https://www.chessprogramming.org/Selectivity should be a starting point, otherwise you can maybe ask it as a new question.  
  > **username_0**: Makes sense. I figured the pruning would only happen when the engine is near the depth limit, since if a move doesn't have a high evaluation at that point it's most likely bad.  
  
  ---
  
  