Answers.Body: <p>Disclaimer: not an expert.</p>
  
  <p>As I understand the so-called "<a href="http://tests.stockfishchess.org/tests" rel="nofollow noreferrer">Fishtesting</a>", you write a patch and submit it to the queue. The system then plays your patch against the previous engine at short time controls. Very many games are played (this is why computing power is needed to improve your engine). If the patch proves to be stronger at short time controls, another test is performed and longer time controls, and if the patch passes again then it's promoted to become the main version.</p>
  
  <p>Example: <a href="http://tests.stockfishchess.org/html/live_elo.html?5c4f854b0ebc593af5d4cd72" rel="nofollow noreferrer">this patch</a>. You can see the user submitting the patch is "protonspring", and he was trying to simplify the passed pawns code. Fishtesting played the new patch against the old version 5451 times, with a result of 20.1% wins, 23.3% losses, and 56.5% draws. The new patch is inferior, so it's rejected. As you can see from the Fishtest main page, most patches fail.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-29T18:45:20.090Z
Answers.Text: So it seems like my testing method is conceptually correct, but it needs greater computational power to run more trials and be more effective. On my PC I'm only able to run 200-400 trials overnight, with my engines thinking at 1 second/move.
Answers.UserDisplayName: Inertial Ignorance
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-29T00:56:31.773Z
Answers.LastActivityDate: 2019-01-29T00:56:31.773Z
Answers.UserDisplayName: Allure
Body: <p>Does Stockfish's testing framework get the new engine to play the old engine many times in a match? Since Stockfish thinks on each move according to a time limit (and not a depth limit, which gives a deterministic amount of calculation), randomness in the OS' speed affects the engine's performance. This means that the match result could be more determined by randomness in the OS than by the incremental change made to Stockfish, causing the new engine to lose.</p>
  
  <p>I'm working on my own engine with this testing framework, and I've found this to be a problem. Are there any other testing frameworks for engines that aren't subject to how the OS is performing? I also run matches with a depth limit for the engines, but these matches aren't ideal because in some positions an engine needs to calculate more or can get away with calculating less.</p>
  
Tags: <engines><stockfish>
Title: How does Stockfish ensure an incremental change made is beneficial?
ThreadText: # How does Stockfish ensure an incremental change made is beneficial?
  
  (Asked by: username_0 on 2019-01-28)
  
  Does Stockfish's testing framework get the new engine to play the old engine many times in a match? Since Stockfish thinks on each move according to a time limit (and not a depth limit, which gives a deterministic amount of calculation), randomness in the OS' speed affects the engine's performance. This means that the match result could be more determined by randomness in the OS than by the incremental change made to Stockfish, causing the new engine to lose.
  
  I'm working on my own engine with this testing framework, and I've found this to be a problem. Are there any other testing frameworks for engines that aren't subject to how the OS is performing? I also run matches with a depth limit for the engines, but these matches aren't ideal because in some positions an engine needs to calculate more or can get away with calculating less.
  
  **Tags:** engines, stockfish
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-01-29. Score: 3 | Accepted answer
  
  Disclaimer: not an expert.
  
  As I understand the so-called "[Fishtesting](http://tests.stockfishchess.org/tests)", you write a patch and submit it to the queue. The system then plays your patch against the previous engine at short time controls. Very many games are played (this is why computing power is needed to improve your engine). If the patch proves to be stronger at short time controls, another test is performed and longer time controls, and if the patch passes again then it's promoted to become the main version.
  
  Example: [this patch](http://tests.stockfishchess.org/html/live_elo.html?5c4f854b0ebc593af5d4cd72). You can see the user submitting the patch is "protonspring", and he was trying to simplify the passed pawns code. Fishtesting played the new patch against the old version 5451 times, with a result of 20.1% wins, 23.3% losses, and 56.5% draws. The new patch is inferior, so it's rejected. As you can see from the Fishtest main page, most patches fail.
  
  > **Comments:**
  > **username_0**: So it seems like my testing method is conceptually correct, but it needs greater computational power to run more trials and be more effective. On my PC I'm only able to run 200-400 trials overnight, with my engines thinking at 1 second/move.  
  
  ---
  
  