Answers.Body: <p>There's not always a clear-cut point at which a game deviates from theory. A common use of the term refers to the first move that hasn't been played before, and as 10...Qe5 is the first move that doesn't appear on Lichess masters' database, we could say that move is a &quot;novelty&quot; according to the reference database we're using. Most serious chess players would agree with this assessment.</p>
  <p>If we consider all games ever played on Lichess, then 12...Qb2 is indeed the &quot;novelty&quot;, but most people wouldn't consider that every online Blitz game is a novelty. However for this specific use case it may be more useful to consider &quot;theory&quot; ends here, as there's no reason to expect ChatGPT's training set to prefer master games over amateur games.</p>
  <p>Generative AI is basically glorified autocomplete so it's perfectly possible it was aware of a chess game starting with that line, but that would beg the question of why it chose that specific line instead of some other more common variation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-14T11:24:38.907Z
Answers.Text: I'll wait a bit whether someone else comes up with a game from another database before accepting the answer. I do think master games would be much more likely to make it into the training of GPT models, because master games (and more generally, serious games) are likely to be preserved in various sources, making it more likely that they are in a source that has been scraped for training data than random internet games.
Answers.UserDisplayName: Polytropos
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-14T11:29:41.627Z
Answers.Text: @Polytropos you may want to check on YottaBase
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-14T07:46:31.200Z
Answers.LastActivityDate: 2023-04-14T07:46:31.200Z
Answers.UserDisplayName: David
Body: <p>I have recently been tinkering on and off with ways to make ChatGPT play chess. This has worked somewhat well, even with ChatGPT3.5, using a prompt crafted to make ChatGPT re-prompt itself at every turn with the full game history so far. I would estimate that it played at maybe 1000-1100 elo level, whereas GPT-4 reached maybe 1400. By computer chess standards that is obviously abysmal, but I'm fascinated with it as this thing is not meant to play chess.</p>
  <p>My most recent attempt with this little pastime involved augmenting my usual prompt with a single, chosen to be instructive example of good play. Concretely, I gave it Morphy's Opera Game and asked it to emulate White's playstyle. The exact prompt and game can be found <a href="https://lichess.org/study/ymmMxzbj/kKbpLLn8" rel="nofollow noreferrer">here (i.e. chapter 15 in the study)</a>.</p>
  <p>In my previous experiments, coherence of play was mostly lost when the sum total of all prompting and self-prompting during a game ongoing in this manner reached the size of the context window (but this could be remedied by reprompting with only the game history, i.e. removing chat history). Without having counted tokens, I expect that this was the same here, with time to decoherence shortened slightly due to the Morphy game prompt.</p>
  <p>However, up to that point, I was <em>totally blown away</em> by the quality of ChatGPT's play in this game.</p>
  <pre><code>[FEN &quot;&quot;]
  [White &quot;ChatGPT 3.5&quot;]
  [Black &quot;Shredder Chess Online (level hard)&quot;]
  [Result &quot;0-1&quot;]
  [UTCDate &quot;2023.04.13&quot;]
  [Variant &quot;Standard&quot;]
  [ECO &quot;C45&quot;]
  [Opening &quot;Scotch Game: Modern Defense&quot;]
  
  1. e4 { [%eval 0.36] } { Prompt was:
  We are playing a chess game. At each turn, answer in a code box strictly in the following format:
  PGN of the game so far:
  Best move for White:
  ---
  Here is an example game where White played well:
  (...followed by a commented PGN of the famous Opera Game by Morphy...)
  Try to play like White did in that game. } 1... e5 { [%eval 0.35] } 2. Nf3 { [%eval 0.29] } 2... Nc6 { [%eval 0.32] } 3. d4 { [%eval 0.13] } 3... exd4 { [%eval 0.13] } 4. Nxd4 { [%eval 0.03] } 4... Qh4 { [%eval 0.57] } 5. Nc3 { [%eval 0.95] } 5... Bb4 { [%eval 0.87] } 6. Be2 { [%eval 0.84] } 6... Bxc3+ { [%eval 0.93] } 7. bxc3 { [%eval 0.94] } 7... Nf6? { [%eval 2.22] } { Mistake. Nxd4 was best. } (7... Nxd4 8. Qxd4 Ne7 9. c4 O-O 10. Bb2 Qh6 11. f4 d6 12. O-O) 8. O-O? { [%eval 0.55] } { Mistake. Nf5 was best. } (8. Nf5 Qxe4 9. Nxg7+ Kd8 10. O-O Qg6 11. Nh5 Rg8 12. Ng3 d6) 8... O-O?! { [%eval 1.14] } { Inaccuracy. Nxd4 was best. } (8... Nxd4) 9. Nf5 { [%eval 0.85] } 9... Qxe4 { [%eval 1.28] } 10. Bd3 { [%eval 1.11] } 10... Qe5 { [%eval 1.18] } 11. Re1 { [%eval 1.49] } 11... Qxc3? { [%eval 3.22] } { Mistake. Qa5 was best. } (11... Qa5 12. Rb1) 12. Bd2?! { [%eval 2.42] } { leandroac - FreiwiedieGeier, 0-1, -, https://lichess.org/yD80NGqu } (12. Rb1 d6 13. Bb2 Qc5 14. Nh6+ Kh8 15. Bxf6 gxf6 16. Rb5 Qc3 17. Rh5 Ne5 18. Bxh7 Re8) 12... Qb2? { [%eval 4.49] } { Mistake. Qc5 was best. } (12... Qc5) 13. Rb1 { [%eval 4.61] } 13... Qxa2 { [%eval 5.61] } 14. Bc3 { [%eval 5.39] } 14... d6 { [%eval 4.64] } 15. Bxf6 { [%eval 4.14] } 15... Bxf5 { [%eval 4.1] } 16. Bxf5 { [%eval 4.17] } 16... Qc4?! { [%eval 5.77] } { Inaccuracy. Ne5 was best. } (16... Ne5 17. Bxe5 dxe5 18. Rxb7 g6 19. Bd3 Rfd8 20. Qa1 Qxa1 21. Rxa1 a5 22. Rxc7 a4 23. h3) 17. Re4 { [%eval 5.6] } (17. Qh5 h6 18. Re4 Qa2 19. Rf1 Ne5 20. Rg4 Ng6 21. Bxg7)  (17. Bxh7+ Kxh7 18. Qh5+ Kg8 19. Qg5) 17... Nd4?! { [%eval 9.16] } { Inaccuracy. Qa2 was best. } (17... Qa2 18. Ra1 Qxa1 19. Qxa1 g6 20. Bd7 Ne5 21. Bxe5 Rfe8 22. Bh8 Re5 23. Bxe5 dxe5 24. Qxe5) 18. Rxd4 { [%eval 9.16] } 18... Qa2? { [%eval #9] } { Checkmate is now unavoidable. g6 was best. } (18... g6 19. Rxc4 Rae8 20. Rxc7 Re5 21. Bxe5 dxe5 22. Be4 Kh8 23. Qd5 g5 24. Rcxb7) 19. Bxh7+ { [%eval #7] } 19... Kxh7 { [%eval #7] } 20. Rh4+?? { [%eval 6.59] } { Lost forced checkmate sequence. Rg4 was best. } (20. Rg4 Qxb1 21. Rxg7+ Kh8 22. Rg4+ Kh7 23. Qxb1 Rg8 24. c3+ Rg6 25. Rh4+ Kg8 26. Rh8#) 20... Kg6 { [%eval 6.69] } 21. Qh5+?! { [%eval 5.1] } { Inaccuracy. Bc3 was best. } (21. Bc3 Qxb1 22. Qxb1 f5 23. Qb5 Rae8 24. Qd7 Re5 25. Rh3 f4 26. Rh4 b5 27. Qg4+ Rg5) 21... Kxf6 { [%eval 5.29] } 22. Re1 { [%eval 5.16] } 22... g6 { [%eval 4.72] } 23. Rf4+ Kg7 { Here, the game was adjourned, because ChatGPT3.5 lost track of the position. } 0-1
  
  
  
  </code></pre>
  <p>This of course begs the question whether the game could have been in its training data. The lichess database suggests that the game departed from other games in the database when Shredder Chess Online played 12. ... Qb2.</p>
  <p>Are there any other games anywhere that follow this game for longer, and if so, what is the source?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-15T09:24:26.087Z
Comments.Text: impressive that you found a way to make it play legal moves. According to videos I watched by Hikaru Nakamura and Levy Rozman, when they played ChatGPT, it kept suggesting illegal moves.
Comments.UserDisplayName: Chris Sanders
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-21T18:33:20.830Z
Comments.Text: @ChrisSanders [OpenAI's new language model gpt-3.5-turbo-instruct can defeat chess engine Fairy-Stockfish 14 at level 5](https://redd.it/16oi6fb)
Comments.UserDisplayName: Franck Dernoncourt
Tags: <opening><theory><computer-chess>
Title: When did this chess game against ChatGPT diverge from theory?
ThreadText: # When did this chess game against ChatGPT diverge from theory?
  
  (Asked by: username_0 on 2023-04-13)
  
  I have recently been tinkering on and off with ways to make ChatGPT play chess. This has worked somewhat well, even with ChatGPT3.5, using a prompt crafted to make ChatGPT re-prompt itself at every turn with the full game history so far. I would estimate that it played at maybe 1000-1100 elo level, whereas GPT-4 reached maybe 1400. By computer chess standards that is obviously abysmal, but I'm fascinated with it as this thing is not meant to play chess.
  
  My most recent attempt with this little pastime involved augmenting my usual prompt with a single, chosen to be instructive example of good play. Concretely, I gave it Morphy's Opera Game and asked it to emulate White's playstyle. The exact prompt and game can be found [here (i.e. chapter 15 in the study)](https://lichess.org/study/ymmMxzbj/kKbpLLn8).
  
  In my previous experiments, coherence of play was mostly lost when the sum total of all prompting and self-prompting during a game ongoing in this manner reached the size of the context window (but this could be remedied by reprompting with only the game history, i.e. removing chat history). Without having counted tokens, I expect that this was the same here, with time to decoherence shortened slightly due to the Morphy game prompt.
  
  However, up to that point, I was *totally blown away* by the quality of ChatGPT's play in this game.
  
  ```
  [FEN ""]
  [White "ChatGPT 3.5"]
  [Black "Shredder Chess Online (level hard)"]
  [Result "0-1"]
  [UTCDate "2023.04.13"]
  [Variant "Standard"]
  [ECO "C45"]
  [Opening "Scotch Game: Modern Defense"]
  
  1. e4 { [%eval 0.36] } { Prompt was:
  We are playing a chess game. At each turn, answer in a code box strictly in the following format:
  PGN of the game so far:
  Best move for White:
  ---
  Here is an example game where White played well:
  (...followed by a commented PGN of the famous Opera Game by Morphy...)
  Try to play like White did in that game. } 1... e5 { [%eval 0.35] } 2. Nf3 { [%eval 0.29] } 2... Nc6 { [%eval 0.32] } 3. d4 { [%eval 0.13] } 3... exd4 { [%eval 0.13] } 4. Nxd4 { [%eval 0.03] } 4... Qh4 { [%eval 0.57] } 5. Nc3 { [%eval 0.95] } 5... Bb4 { [%eval 0.87] } 6. Be2 { [%eval 0.84] } 6... Bxc3+ { [%eval 0.93] } 7. bxc3 { [%eval 0.94] } 7... Nf6? { [%eval 2.22] } { Mistake. Nxd4 was best. } (7... Nxd4 8. Qxd4 Ne7 9. c4 O-O 10. Bb2 Qh6 11. f4 d6 12. O-O) 8. O-O? { [%eval 0.55] } { Mistake. Nf5 was best. } (8. Nf5 Qxe4 9. Nxg7+ Kd8 10. O-O Qg6 11. Nh5 Rg8 12. Ng3 d6) 8... O-O?! { [%eval 1.14] } { Inaccuracy. Nxd4 was best. } (8... Nxd4) 9. Nf5 { [%eval 0.85] } 9... Qxe4 { [%eval 1.28] } 10. Bd3 { [%eval 1.11] } 10... Qe5 { [%eval 1.18] } 11. Re1 { [%eval 1.49] } 11... Qxc3? { [%eval 3.22] } { Mistake. Qa5 was best. } (11... Qa5 12. Rb1) 12. Bd2?! { [%eval 2.42] } { leandroac - FreiwiedieGeier, 0-1, -, https://lichess.org/yD80NGqu } (12. Rb1 d6 13. Bb2 Qc5 14. Nh6+ Kh8 15. Bxf6 gxf6 16. Rb5 Qc3 17. Rh5 Ne5 18. Bxh7 Re8) 12... Qb2? { [%eval 4.49] } { Mistake. Qc5 was best. } (12... Qc5) 13. Rb1 { [%eval 4.61] } 13... Qxa2 { [%eval 5.61] } 14. Bc3 { [%eval 5.39] } 14... d6 { [%eval 4.64] } 15. Bxf6 { [%eval 4.14] } 15... Bxf5 { [%eval 4.1] } 16. Bxf5 { [%eval 4.17] } 16... Qc4?! { [%eval 5.77] } { Inaccuracy. Ne5 was best. } (16... Ne5 17. Bxe5 dxe5 18. Rxb7 g6 19. Bd3 Rfd8 20. Qa1 Qxa1 21. Rxa1 a5 22. Rxc7 a4 23. h3) 17. Re4 { [%eval 5.6] } (17. Qh5 h6 18. Re4 Qa2 19. Rf1 Ne5 20. Rg4 Ng6 21. Bxg7)  (17. Bxh7+ Kxh7 18. Qh5+ Kg8 19. Qg5) 17... Nd4?! { [%eval 9.16] } { Inaccuracy. Qa2 was best. } (17... Qa2 18. Ra1 Qxa1 19. Qxa1 g6 20. Bd7 Ne5 21. Bxe5 Rfe8 22. Bh8 Re5 23. Bxe5 dxe5 24. Qxe5) 18. Rxd4 { [%eval 9.16] } 18... Qa2? { [%eval #9] } { Checkmate is now unavoidable. g6 was best. } (18... g6 19. Rxc4 Rae8 20. Rxc7 Re5 21. Bxe5 dxe5 22. Be4 Kh8 23. Qd5 g5 24. Rcxb7) 19. Bxh7+ { [%eval #7] } 19... Kxh7 { [%eval #7] } 20. Rh4+?? { [%eval 6.59] } { Lost forced checkmate sequence. Rg4 was best. } (20. Rg4 Qxb1 21. Rxg7+ Kh8 22. Rg4+ Kh7 23. Qxb1 Rg8 24. c3+ Rg6 25. Rh4+ Kg8 26. Rh8#) 20... Kg6 { [%eval 6.69] } 21. Qh5+?! { [%eval 5.1] } { Inaccuracy. Bc3 was best. } (21. Bc3 Qxb1 22. Qxb1 f5 23. Qb5 Rae8 24. Qd7 Re5 25. Rh3 f4 26. Rh4 b5 27. Qg4+ Rg5) 21... Kxf6 { [%eval 5.29] } 22. Re1 { [%eval 5.16] } 22... g6 { [%eval 4.72] } 23. Rf4+ Kg7 { Here, the game was adjourned, because ChatGPT3.5 lost track of the position. } 0-1
  
  ```
  
  This of course begs the question whether the game could have been in its training data. The lichess database suggests that the game departed from other games in the database when Shredder Chess Online played 12. ... Qb2.
  
  Are there any other games anywhere that follow this game for longer, and if so, what is the source?
  
  **Tags:** opening, theory, computer-chess
  
  > **Comments:**
  > **username_1**: impressive that you found a way to make it play legal moves. According to videos I watched by Hikaru Nakamura and Levy Rozman, when they played ChatGPT, it kept suggesting illegal moves.  
  > **username_2**: @username_1 [OpenAI's new language model gpt-3.5-turbo-instruct can defeat chess engine Fairy-Stockfish 14 at level 5](https://redd.it/16oi6fb)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2023-04-14. Score: 4 | Accepted answer
  
  There's not always a clear-cut point at which a game deviates from theory. A common use of the term refers to the first move that hasn't been played before, and as 10...Qe5 is the first move that doesn't appear on Lichess masters' database, we could say that move is a "novelty" according to the reference database we're using. Most serious chess players would agree with this assessment.
  
  If we consider all games ever played on Lichess, then 12...Qb2 is indeed the "novelty", but most people wouldn't consider that every online Blitz game is a novelty. However for this specific use case it may be more useful to consider "theory" ends here, as there's no reason to expect ChatGPT's training set to prefer master games over amateur games.
  
  Generative AI is basically glorified autocomplete so it's perfectly possible it was aware of a chess game starting with that line, but that would beg the question of why it chose that specific line instead of some other more common variation.
  
  > **Comments:**
  > **username_0**: I'll wait a bit whether someone else comes up with a game from another database before accepting the answer. I do think master games would be much more likely to make it into the training of GPT models, because master games (and more generally, serious games) are likely to be preserved in various sources, making it more likely that they are in a source that has been scraped for training data than random internet games.  
  > **username_3**: @username_0 you may want to check on YottaBase  
  
  ---
  
  