Answers.Body: <p>Michael West is right that the sample size is small.</p>
  <blockquote>
  <p>I do not consider these numbers, each player with 80 games (16 between every pair), too small for discerning some signal above the natural noise level.</p>
  </blockquote>
  <p>Check <a href="https://pastebin.com/dz0PZymr" rel="nofollow noreferrer">this</a> out, it's from TCEC small sample size tests. You get the <em>same engine</em> to play 100 games against itself, in 1000 matches (so, 100,000 games). Note a 65-35 score happens twice. That's a +30 point difference, even though it's the <em>same engine</em> playing both sides. That's small sample size for you.</p>
  <p>There is a reason engine devs need tens of thousands of games to prove a patch is superior.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-06T14:30:17.250Z
Answers.LastActivityDate: 2023-11-06T14:30:17.250Z
Answers.UserDisplayName: Allure
Body: <p>I have run computer chess tournaments among Stockfish engines, set up for various nominal strengths. The results were really strange, some are in reverse order to their supposed performance. For example, here is a crosstable (8x2 rounds) for Skill Levels 4-7, as well as two UCI-Elo settings of 1700 and 1900:</p>
  <pre><code> Rank                                     Name    1     2     3     4     5     6  Games  Score  Score%  
  1      Stockfish_Ver16_Skill=7_st=0.100sec    x  12.0  12.0   8.5  13.5  14.5     80   60.5   75.62   
  2      Stockfish_Ver16_Skill=6_st=0.100sec  4.0     x  11.0  11.0  13.0  14.5     80   53.5   66.88   
  3      Stockfish_Ver16_Skill=5_st=0.100sec  4.0   5.0     x  12.5  11.0  13.0     80   45.5   56.88   
  4 Stockfish_Ver16_UCI_Elo=1900_tc=40/15min  7.5   5.0   3.5     x  10.0  11.0     80   37.0   46.25   
  5      Stockfish_Ver16_Skill=4_st=0.100sec  2.5   3.0   5.0   6.0     x   5.5     80   22.0   27.50   
  6 Stockfish_Ver16_UCI_Elo=1700_tc=40/15min  1.5   1.5   3.0   5.0  10.5     x     80   21.5   26.88   
  </code></pre>
  <p>So, the 4th ranked had quite good 7.5:8.5 showing against the 1st, and progressively worse against the 2nd and 3rd!
  How could this be?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-04T03:40:56.830Z
Comments.Text: Perhaps because the number of games is small.
Comments.UserDisplayName: Michael West
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-05T08:11:00.980Z
Comments.Text: @MichaelWest > Perhaps because the number of games is small. 
  
   OFC more is always better for S/N. But I do not consider these numbers, each player with 80 games (16 between every pair), too small for discerning some signal above the natural noise level.
Comments.UserDisplayName: Ch3cks Out
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-05T09:24:03.600Z
Comments.Text: @user30536 > You mention a 'performance pattern', but you don't describe it.  
   The expected pattern is non-increasing scores against a set of opponents, in their order of decreasing strength;
  e.g., in the particular subset I mentioned in my post:  
    
   Score(#4-vs-#1)<=Score(#4-vs-#2)<=Score(#4-vs-#3)  
    
  Instead (as I pointed out), the opposite is observed here, i.e. 7.5>5.0>3.5!
Comments.UserDisplayName: Ch3cks Out
Tags: <stockfish><computer-chess><performance>
Title: What could explain strange performance pattern among Stockfish engine variants?
ThreadText: # What could explain strange performance pattern among Stockfish engine variants?
  
  (Asked by: username_0 on 2023-11-03)
  
  I have run computer chess tournaments among Stockfish engines, set up for various nominal strengths. The results were really strange, some are in reverse order to their supposed performance. For example, here is a crosstable (8x2 rounds) for Skill Levels 4-7, as well as two UCI-Elo settings of 1700 and 1900:
   Rank                                     Name    1     2     3     4     5     6  Games  Score  Score%  
  1      Stockfish_Ver16_Skill=7_st=0.100sec    x  12.0  12.0   8.5  13.5  14.5     80   60.5   75.62   
  2      Stockfish_Ver16_Skill=6_st=0.100sec  4.0     x  11.0  11.0  13.0  14.5     80   53.5   66.88   
  3      Stockfish_Ver16_Skill=5_st=0.100sec  4.0   5.0     x  12.5  11.0  13.0     80   45.5   56.88   
  4 Stockfish_Ver16_UCI_Elo=1900_tc=40/15min  7.5   5.0   3.5     x  10.0  11.0     80   37.0   46.25   
  5      Stockfish_Ver16_Skill=4_st=0.100sec  2.5   3.0   5.0   6.0     x   5.5     80   22.0   27.50   
  6 Stockfish_Ver16_UCI_Elo=1700_tc=40/15min  1.5   1.5   3.0   5.0  10.5     x     80   21.5   26.88   
  
  So, the 4th ranked had quite good 7.5:8.5 showing against the 1st, and progressively worse against the 2nd and 3rd!
  How could this be?
  
  **Tags:** stockfish, computer-chess, performance
  
  > **Comments:**
  > **username_1**: Perhaps because the number of games is small.  
  > **username_0**: @MichaelWest > Perhaps because the number of games is small.    OFC more is always better for S/N. But I do not consider these numbers, each player with 80 games (16 between every pair), too small for discerning some signal above the natural noise level.  
  > **username_0**: @user30536 > You mention a 'performance pattern', but you don't describe it.    The expected pattern is non-increasing scores against a set of opponents, in their order of decreasing strength; e.g., in the particular subset I mentioned in my post:       Score(#4-vs-#1)<=Score(#4-vs-#2)<=Score(#4-vs-#3)      Instead (as I pointed out), the opposite is observed here, i.e. 7.5>5.0>3.5!  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-11-06. Score: 0
  
  username_1 is right that the sample size is small.
  
  > I do not consider these numbers, each player with 80 games (16 between every pair), too small for discerning some signal above the natural noise level.
  
  Check [this](https://pastebin.com/dz0PZymr) out, it's from TCEC small sample size tests. You get the *same engine* to play 100 games against itself, in 1000 matches (so, 100,000 games). Note a 65-35 score happens twice. That's a +30 point difference, even though it's the *same engine* playing both sides. That's small sample size for you.
  
  There is a reason engine devs need tens of thousands of games to prove a patch is superior.
  
  
  ---
  
  