Answers.Body: <p>Since I could not find any complete explanation, I followed the eternal advice &quot;Use the source, Luke&quot;. The Stockfish authors did a great job because their code is easily followable and commented. Relevant parts from <a href="https://github.com/official-stockfish/Stockfish/blob/master/src/search.cpp" rel="noreferrer"><code>search.cpp</code></a>:</p>
  <pre><code>  // If skill level is enabled, swap best PV line with the sub-optimal one
    if (skill.enabled())
        std::swap(rootMoves[0], *std::find(rootMoves.begin(), rootMoves.end(),
                  skill.best ? skill.best : skill.pick_best(multiPV)));
  </code></pre>
  <pre><code>      // If skill level is enabled and time is up, pick a sub-optimal best move
        if (skill.enabled() &amp;&amp; skill.time_to_pick(rootDepth))
            skill.pick_best(multiPV);
  </code></pre>
  <p>And this is the full algorithm used. Here's a summary: the algorithm considers the top-k moves, where k is <code>multiPV</code>, an option chosen by the user and set to at least 4 in the code. <code>delta</code> is computed as the score difference between the best move, ordered first, and the kth best move, however this difference is capped at the value of a pawn to ensure it is not too large. Then, each move is strengthened (has its score increased) by an amount depending on weakness and a random scaling of <code>delta</code>, with <code>% weakness</code> limiting how much effect <code>delta</code> has. If there are many good moves, <code>delta</code> will be small, so each move receives a smaller score push. The algorithm finally chooses the strongest move after these score perturbations.</p>
  <pre><code>  // When playing with strength handicap, choose best move among a set of RootMoves
    // using a statistical rule dependent on 'level'. Idea by Heinz van Saanen.
  
    Move Skill::pick_best(size_t multiPV) {
  
      const RootMoves&amp; rootMoves = Threads.main()-&gt;rootMoves;
      static PRNG rng(now()); // PRNG sequence should be non-deterministic
  
      // RootMoves are already sorted by score in descending order
      Value topScore = rootMoves[0].score;
      int delta = std::min(topScore - rootMoves[multiPV - 1].score, PawnValueMg);
      int weakness = 120 - 2 * level;
      int maxScore = -VALUE_INFINITE;
  
      // Choose best move. For each move score we add two terms, both dependent on
      // weakness. One is deterministic and bigger for weaker levels, and one is
      // random. Then we choose the move with the resulting highest score.
      for (size_t i = 0; i &lt; multiPV; ++i)
      {
          // This is our magic formula
          int push = (  weakness * int(topScore - rootMoves[i].score)
                      + delta * (rng.rand&lt;unsigned&gt;() % weakness)) / 128;
  
          if (rootMoves[i].score + push &gt;= maxScore)
          {
              maxScore = rootMoves[i].score + push;
              best = rootMoves[i].pv[0];
          }
      }
  
      return best;
    }
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-13T12:21:10.677Z
Answers.Text: +1 for the “saying”!
Answers.UserDisplayName: fartgeek
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-18T17:08:50.850Z
Answers.Text: Could you conceptually unpack the part of the algorithm relating to `delta` a little please? (assuming you understood this part of the code better than I did!) It's clearly the random component but I didn't really understand `std::min`, the comparison to `PawnValueMg`, or the `% weakness`...
Answers.UserDisplayName: Mobeus Zoom
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-18T17:47:40.270Z
Answers.Text: @MobeusZoom I added an explanation based on my understanding to the answer.
Answers.UserDisplayName: qwr
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-18T18:08:38.567Z
Answers.Text: Very illuminating, thank you! So it turns out that the weaker the player, (i) the more how much it likes a move is weighted by how bad it is (within the top k); (ii) the more it is prone to randomness (since I guess `% weakness` leaves a larger number as `weakness` increases). What about choosing k? Also, I'd be fascinated if anyone can consider the first part of your original question: why does this lead to the observed alternation between strong play and sudden blunders?
Answers.UserDisplayName: Mobeus Zoom
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-18T20:07:59.750Z
Answers.Text: Yes, the deterministic part makes the move scores closer together (harder to distinguish), and weakness also randomly boosts move scores by a random quantity between 0 and k-1 (then divided by 128).
Answers.UserDisplayName: qwr
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-18T20:11:17.027Z
Answers.Text: I added a note about k. As for blunders, I see the computer make obvious blunders, but I guess at a beginner level like I am it's hard to notice your own blunders until it's too late.
Answers.UserDisplayName: qwr
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-12T22:29:06.853Z
Answers.LastActivityDate: 2021-06-18T20:08:42.697Z
Answers.LastEditDate: 2021-06-18T20:08:42.697Z
Answers.UserDisplayName: qwr
Body: <p>I read on some Lichess forum that Stockfish levels use limited computation time and search depth for lower levels. However, it seems like Stockfish also every so often inputs mistakes and blunders, because it makes some blunders a human, even a beginner, would almost certainly not make. So how are levels implemented?</p>
  
Tags: <stockfish><blunder>
Title: How do Stockfish levels work?
ThreadText: # How do Stockfish levels work?
  
  (Asked by: username_0 on 2021-06-12)
  
  I read on some Lichess forum that Stockfish levels use limited computation time and search depth for lower levels. However, it seems like Stockfish also every so often inputs mistakes and blunders, because it makes some blunders a human, even a beginner, would almost certainly not make. So how are levels implemented?
  
  **Tags:** stockfish, blunder
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2021-06-12. Score: 10 | Accepted answer
  
  Since I could not find any complete explanation, I followed the eternal advice "Use the source, Luke". The Stockfish authors did a great job because their code is easily followable and commented. Relevant parts from [`search.cpp`](https://github.com/official-stockfish/Stockfish/blob/master/src/search.cpp):
  
  ```
    // If skill level is enabled, swap best PV line with the sub-optimal one
    if (skill.enabled())
        std::swap(rootMoves[0], *std::find(rootMoves.begin(), rootMoves.end(),
                  skill.best ? skill.best : skill.pick_best(multiPV)));
  
  ```
  
  ```
        // If skill level is enabled and time is up, pick a sub-optimal best move
        if (skill.enabled() && skill.time_to_pick(rootDepth))
            skill.pick_best(multiPV);
  
  ```
  
  And this is the full algorithm used. Here's a summary: the algorithm considers the top-k moves, where k is `multiPV`, an option chosen by the user and set to at least 4 in the code. `delta` is computed as the score difference between the best move, ordered first, and the kth best move, however this difference is capped at the value of a pawn to ensure it is not too large. Then, each move is strengthened (has its score increased) by an amount depending on weakness and a random scaling of `delta`, with `% weakness` limiting how much effect `delta` has. If there are many good moves, `delta` will be small, so each move receives a smaller score push. The algorithm finally chooses the strongest move after these score perturbations.
  
  ```
    // When playing with strength handicap, choose best move among a set of RootMoves
    // using a statistical rule dependent on 'level'. Idea by Heinz van Saanen.
  
    Move Skill::pick_best(size_t multiPV) {
  
      const RootMoves& rootMoves = Threads.main()->rootMoves;
      static PRNG rng(now()); // PRNG sequence should be non-deterministic
  
      // RootMoves are already sorted by score in descending order
      Value topScore = rootMoves[0].score;
      int delta = std::min(topScore - rootMoves[multiPV - 1].score, PawnValueMg);
      int weakness = 120 - 2 * level;
      int maxScore = -VALUE_INFINITE;
  
      // Choose best move. For each move score we add two terms, both dependent on
      // weakness. One is deterministic and bigger for weaker levels, and one is
      // random. Then we choose the move with the resulting highest score.
      for (size_t i = 0; i < multiPV; ++i)
      {
          // This is our magic formula
          int push = (  weakness * int(topScore - rootMoves[i].score)
                      + delta * (rng.rand<unsigned>() % weakness)) / 128;
  
          if (rootMoves[i].score + push >= maxScore)
          {
              maxScore = rootMoves[i].score + push;
              best = rootMoves[i].pv[0];
          }
      }
  
      return best;
    }
  
  ```
  
  > **Comments:**
  > **username_1**: +1 for the “saying”!  
  > **username_2**: Could you conceptually unpack the part of the algorithm relating to `delta` a little please? (assuming you understood this part of the code better than I did!) It's clearly the random component but I didn't really understand `std::min`, the comparison to `PawnValueMg`, or the `% weakness`...  
  > **username_0**: @username_2 I added an explanation based on my understanding to the answer.  
  > **username_2**: Very illuminating, thank you! So it turns out that the weaker the player, (i) the more how much it likes a move is weighted by how bad it is (within the top k); (ii) the more it is prone to randomness (since I guess `% weakness` leaves a larger number as `weakness` increases). What about choosing k? Also, I'd be fascinated if anyone can consider the first part of your original question: why does this lead to the observed alternation between strong play and sudden blunders?  
  > **username_0**: Yes, the deterministic part makes the move scores closer together (harder to distinguish), and weakness also randomly boosts move scores by a random quantity between 0 and k-1 (then divided by 128).  
  > **username_0**: I added a note about k. As for blunders, I see the computer make obvious blunders, but I guess at a beginner level like I am it's hard to notice your own blunders until it's too late.  
  
  ---
  
  