Answers.Body: <p>If you are not tied to using PythonChess and simply want FEN encodings of each position, you might like to consider using my free, open-source <a href="https://www.cs.kent.ac.uk/%7Edjb/pgn-extract/" rel="noreferrer">pgn-extract</a> program with its -Wepd option to output in EPD format. It is a compiled C program so likely to be significantly faster for this use case. For instance:</p>
  <pre><code>pgn-extract --quiet -Wepd games.pgn
  </code></pre>
  <p>Here are the first few lines of its output for one of the games quoted in the OP:</p>
  <pre><code>rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - c0 sandstorm00-yendorzerep Rated Blitz game 2020.06.01; c1 1/2-1/2;
  rnbqkbnr/pppppppp/8/8/4P3/8/PPPP1PPP/RNBQKBNR b KQkq e3 c0 sandstorm00-yendorzerep Rated Blitz game 2020.06.01; c1 1/2-1/2;
  rnbqkbnr/pppppp1p/6p1/8/4P3/8/PPPP1PPP/RNBQKBNR w KQkq - c0 sandstorm00-yendorzerep Rated Blitz game 2020.06.01; c1 1/2-1/2;
  </code></pre>
  <p>If you wish to suppress irrelevant en passant squares then there is also the --nofauxep option to do that.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-21T20:48:57.553Z
Answers.Text: I compiled and tested pgn-extract by piping the first 1000 games in my database. The FENs were generated in about 1 second. So, yes, I think it's worth making the switch! Even considered writing a python-wrapper for pgn-extract, but unsure if there is enough demand to justify that.
Answers.UserDisplayName: MattY
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-23T06:57:54.747Z
Answers.Text: I am glad that worked for you.
Answers.UserDisplayName: kentdjb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-19T13:50:12.593Z
Answers.LastActivityDate: 2021-07-19T13:50:12.593Z
Answers.UserDisplayName: kentdjb
Answers.Body: <p>As Kevin indicated, you're reevaluating your entire position after every move. A better way to code would be to use the <a href="https://python-chess.readthedocs.io/en/latest/pgn.html?highlight=next#chess.pgn.GameNode.next" rel="nofollow noreferrer"><code>next</code></a> method. Below is a code snippet that should give you an idea:</p>
  <pre><code>import chess.pgn
  pgn = open(&quot;anderssen_kieseritzky_1851.pgn&quot;)
  mygame=chess.pgn.read_game(pgn)
  while mygame.next():
      mygame=mygame.next()
      print(mygame.board().fen())
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-16T02:20:34.420Z
Answers.Text: `AttributeError: 'Game' object has no attribute 'next'`
Answers.UserDisplayName: zfj3ub94rf576hc4eegm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-17T03:50:45.827Z
Answers.Text: next is not an attribute but a function. Use next().
Answers.UserDisplayName: Abhinav S.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-17T15:48:44.747Z
Answers.Text: @zfj3ub94rf576hc4eegm hope that helps
Answers.UserDisplayName: Abhinav S.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-22T06:14:18.503Z
Answers.Text: I think yours is an older version of python-chess. The docs currently recommend iterating over `game.mainline_moves()` https://python-chess.readthedocs.io/en/latest/pgn.html
Answers.UserDisplayName: zfj3ub94rf576hc4eegm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-16T22:02:11.400Z
Answers.Text: @zfj3ub94rf576hc4eegm I am using the latest version. Here's the documentation - https://python-chess.readthedocs.io/en/latest/pgn.html#chess.pgn.GameNode.next
  Can you pls check
Answers.UserDisplayName: Abhinav S.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-20T23:14:38.570Z
Answers.LastActivityDate: 2022-09-20T23:18:42.317Z
Answers.LastEditDate: 2022-09-20T23:18:42.317Z
Answers.UserDisplayName: Abhinav S.
Body: <p>Is there a <strong>fast</strong> way to generate FEN strings for every move in a PGN?</p>
  <p>I'm generating an opening book using the Lichess elite database and Python Chess, but Python Chess is taking a surprisingly long time to analyze games and generate FEN strings for each move. Like, 1/1000th the speed I expected.</p>
  <p>This is how I've done it - it works, but again, slow. <code>board = game.board()</code> is the primary slowdown - but I don't see a way to get FENs directly from the game object.</p>
  <pre><code># PARSE PGNS and create an opening book using a giant dictionary
  
  import chess
  import chess.pgn
  import io
  import datetime
  
  #START WITH SOME DUMMY DATA FOR DEBUGGING
  
  games = '''[Event &quot;Rated Rapid game&quot;]
  [LichessURL &quot;https://lichess.org/YcFqJqyM&quot;]
  [Date &quot;2020.06.01&quot;]
  [Round &quot;-&quot;]
  [White &quot;Poecraft94&quot;]
  [Black &quot;Germanvince&quot;]
  [Result &quot;1-0&quot;]
  [WhiteElo &quot;2323&quot;]
  [BlackElo &quot;2415&quot;]
  [ECO &quot;B50&quot;]
  [Opening &quot;Sicilian Defense&quot;]
  [TimeControl &quot;600+0&quot;]
  [UTCDate &quot;2020.06.01&quot;]
  [UTCTime &quot;00:00:01&quot;]
  [Termination &quot;Normal&quot;]
  [WhiteRatingDiff &quot;+9&quot;]
  [BlackRatingDiff &quot;-8&quot;]
  
  1. e4 c5 2. Nf3 d6 3. Nc3 Nf6 4. g3 g6 5. Bg2 Bg7 6. O-O O-O 7. d4 cxd4 8.
  Nxd4 a6 9. Re1 Nbd7 10. Bg5 Rb8 11. Nd5 e6 12. Nxf6+ Bxf6 13. Bxf6 Nxf6 14.
  Qd2 Qc7 15. Rad1 Kg7 16. Nf3 Rd8 17. e5 Ne8 18. Re3 b5 19. Rd3 d5 20. Rc3
  Qe7 21. Nd4 Bb7 22. Rf3 Rbc8 23. g4 h6 24. Rh3 g5 25. f4 gxf4 26. Rh5 Nc7
  27. Qxf4 Rh8 28. Rf1 Rcg8 29. Rf5 Rf8 30. g5 h5 31. Bh3 b4 32. b3 Nb5 33.
  Rf6 Nxd4 34. Qxd4 Bc6 35. R1f2 Bd7 36. Qa7 Rh7 37. g6 1-0
  
  [Event &quot;Rated Blitz game&quot;]
  [LichessURL &quot;https://lichess.org/AY9kSWFt&quot;]
  [Date &quot;2020.06.01&quot;]
  [Round &quot;-&quot;]
  [White &quot;sandstorm00&quot;]
  [Black &quot;yendorzerep&quot;]
  [Result &quot;1/2-1/2&quot;]
  [WhiteTitle &quot;NM&quot;]
  [BlackTitle &quot;IM&quot;]
  [WhiteElo &quot;2497&quot;]
  [BlackElo &quot;2426&quot;]
  [ECO &quot;B06&quot;]
  [Opening &quot;Modern Defense: Standard Line&quot;]
  [TimeControl &quot;180+0&quot;]
  [UTCDate &quot;2020.06.01&quot;]
  [UTCTime &quot;00:00:19&quot;]
  [Termination &quot;Normal&quot;]
  [WhiteRatingDiff &quot;-1&quot;]
  [BlackRatingDiff &quot;+1&quot;]
  
  1. e4 g6 2. d4 Bg7 3. Nc3 c6 4. Be3 d5 5. e5 f6 6. exf6 Nxf6 7. Bd3 O-O 8.
  h3 Nbd7 9. Nf3 Qc7 10. Qd2 e5 11. dxe5 Nxe5 12. Nxe5 Qxe5 13. O-O-O Bf5 14.
  Bxf5 Qxf5 15. g4 Qf3 16. Rhg1 Ne4 17. Nxe4 Qxe4 18. h4 a5 19. h5 a4 20. a3
  Rae8 21. Rde1 Qe5 22. c3 Qd6 23. f4 Re4 24. hxg6 hxg6 25. f5 Rfe8 26. Bf2
  Rxe1+ 27. Bxe1 d4 28. Bf2 Re4 29. fxg6 Qxg6 30. Bxd4 Bxd4 31. cxd4 Rxg4 32.
  Rxg4 Qxg4 33. Qe3 Kf7 34. Kd2 Qe6 35. Qf4+ Ke7 36. Qc7+ Qd7 37. Qxd7+ Kxd7
  38. Kc3 Kd6 39. Kb4 b5 40. b3 axb3 41. Kxb3 c5 42. dxc5+ Kxc5 43. Kc3 Kc6
  44. Kb4 Kb6 45. a4 bxa4 46. Kxa4 1/2-1/2'''
  
  # If you have the Lichess elite database, yoou can parse the entire thing by uncommenting here
  
  #with open('/lichess_elite_2020-06.pgn') as f:
  #    games = f.read()
  
  games = games.replace(&quot;\n\n[&quot;, &quot;||||[&quot;)
  pgns = games.split(&quot;||||&quot;)
  
  import time
  
  fens = {}
  start = datetime.datetime.utcnow()
  
  for i, pgn in enumerate(pgns):
      game = chess.pgn.read_game(io.StringIO(pgn)) #parse each pgn string
      if game.errors:
          continue
  
      parent_pos = None
      move_num = 0
      
      while game:
          move_num += 1
          if move_num &gt; 40: #practically, we only need to first 20 moves for our book
              break
  
          game=game.next()
          
          if game:
              board = game.board()
              pos = board.fen(en_passant='fen') #grab fen string of current position
              move = game.san() #what was the last move made?
  
              if pos not in fens:
                  fens[pos]={}
  
              if parent_pos:
                  #in the previous position, the current move was made X times
                  if pos in fens[parent_pos]:
                      fens[parent_pos][move] += 1
                  else:
                      fens[parent_pos][move] = 1
  
          parent_pos=pos
  
  print('Run time: ', (datetime.datetime.utcnow()-start).seconds)
  </code></pre>
  <p>This yields a data structure with FEN strings as keys, and the most common replies as values. E.g. after parsing the 2 games, e4 is followed by g6 once and c5 once.</p>
  <pre><code>{
      'rnbqkbnr/pppppppp/8/8/4P3/8/PPPP1PPP/RNBQKBNR b KQkq e3 0 1': {
          'c5': 1,
          'g6': 1}
  ...
  }```
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-18T15:28:06.740Z
Comments.Text: "Fast" is relative: Object-oriented+Python = you see the taillights of me old FORTRAN guy :-) Still, I program in Python a lot and a simple string parsing app should work fast enough (maybe unless you have millions of games), so mind to give some concrete timing values? And structure and example data for your variables? It might work faster the obvious way: starting from RNBQ.../ and parse the moves of the PGN directly, updating the FEN.
Comments.UserDisplayName: Hauke Reddmann
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-18T18:38:56.720Z
Comments.Text: @HaukeReddmann One caveat is that I need the strings to comply with the FEN standard for my downstream use case. That means encoding legal en-passant moves, and so on. So, for that reason, I'd love to NOT generate the FEN strings myself!
Comments.UserDisplayName: MattY
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-18T22:58:50.587Z
Comments.Text: @HaukeReddmann I updated the post with complete code. Note that the script's able to process 100 games every 2 seconds, which means my little database of 400k games will take over 2 hours, and larger databases will take forever.
Comments.UserDisplayName: MattY
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-19T06:37:45.323Z
Comments.Text: I took a look at the documentation of PythonChess. First Impression: Very meticulous, very by-the-book-of-good-programming. But your use case needs speed (100 games/2 seconds sounds slow to me either) and it might be more effective to grin and bear it, throw the nice features of PythonChess out of the window, program from scratch and optimize for speed.
Comments.UserDisplayName: Hauke Reddmann
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-19T22:09:47.157Z
Comments.Text: FYI, `game.board()` is expensive because it is [re-evaluating the entire game from move one](https://python-chess.readthedocs.io/en/latest/_modules/chess/pgn.html#ChildNode) every time you call that method! Since the logic in that method does not look terribly complicated, you could probably rewrite it yourself to iterate over each board in a single pass.
Comments.UserDisplayName: Kevin
Tags: <database><pgn><fen><python>
Title: Quick method for generating FEN strings from PGN using Python Chess
ThreadText: # Quick method for generating FEN strings from PGN using Python Chess
  
  (Asked by: username_0 on 2021-07-18)
  
  Is there a **fast** way to generate FEN strings for every move in a PGN?
  
  I'm generating an opening book using the Lichess elite database and Python Chess, but Python Chess is taking a surprisingly long time to analyze games and generate FEN strings for each move. Like, 1/1000th the speed I expected.
  
  This is how I've done it - it works, but again, slow. `board = game.board()` is the primary slowdown - but I don't see a way to get FENs directly from the game object.
  
  ```
  # PARSE PGNS and create an opening book using a giant dictionary
  
  import chess
  import chess.pgn
  import io
  import datetime
  
  #START WITH SOME DUMMY DATA FOR DEBUGGING
  
  games = '''[Event "Rated Rapid game"]
  [LichessURL "https://lichess.org/YcFqJqyM"]
  [Date "2020.06.01"]
  [Round "-"]
  [White "Poecraft94"]
  [Black "Germanvince"]
  [Result "1-0"]
  [WhiteElo "2323"]
  [BlackElo "2415"]
  [ECO "B50"]
  [Opening "Sicilian Defense"]
  [TimeControl "600+0"]
  [UTCDate "2020.06.01"]
  [UTCTime "00:00:01"]
  [Termination "Normal"]
  [WhiteRatingDiff "+9"]
  [BlackRatingDiff "-8"]
  
  1. e4 c5 2. Nf3 d6 3. Nc3 Nf6 4. g3 g6 5. Bg2 Bg7 6. O-O O-O 7. d4 cxd4 8.
  Nxd4 a6 9. Re1 Nbd7 10. Bg5 Rb8 11. Nd5 e6 12. Nxf6+ Bxf6 13. Bxf6 Nxf6 14.
  Qd2 Qc7 15. Rad1 Kg7 16. Nf3 Rd8 17. e5 Ne8 18. Re3 b5 19. Rd3 d5 20. Rc3
  Qe7 21. Nd4 Bb7 22. Rf3 Rbc8 23. g4 h6 24. Rh3 g5 25. f4 gxf4 26. Rh5 Nc7
  27. Qxf4 Rh8 28. Rf1 Rcg8 29. Rf5 Rf8 30. g5 h5 31. Bh3 b4 32. b3 Nb5 33.
  Rf6 Nxd4 34. Qxd4 Bc6 35. R1f2 Bd7 36. Qa7 Rh7 37. g6 1-0
  
  [Event "Rated Blitz game"]
  [LichessURL "https://lichess.org/AY9kSWFt"]
  [Date "2020.06.01"]
  [Round "-"]
  [White "sandstorm00"]
  [Black "yendorzerep"]
  [Result "1/2-1/2"]
  [WhiteTitle "NM"]
  [BlackTitle "IM"]
  [WhiteElo "2497"]
  [BlackElo "2426"]
  [ECO "B06"]
  [Opening "Modern Defense: Standard Line"]
  [TimeControl "180+0"]
  [UTCDate "2020.06.01"]
  [UTCTime "00:00:19"]
  [Termination "Normal"]
  [WhiteRatingDiff "-1"]
  [BlackRatingDiff "+1"]
  
  1. e4 g6 2. d4 Bg7 3. Nc3 c6 4. Be3 d5 5. e5 f6 6. exf6 Nxf6 7. Bd3 O-O 8.
  h3 Nbd7 9. Nf3 Qc7 10. Qd2 e5 11. dxe5 Nxe5 12. Nxe5 Qxe5 13. O-O-O Bf5 14.
  Bxf5 Qxf5 15. g4 Qf3 16. Rhg1 Ne4 17. Nxe4 Qxe4 18. h4 a5 19. h5 a4 20. a3
  Rae8 21. Rde1 Qe5 22. c3 Qd6 23. f4 Re4 24. hxg6 hxg6 25. f5 Rfe8 26. Bf2
  Rxe1+ 27. Bxe1 d4 28. Bf2 Re4 29. fxg6 Qxg6 30. Bxd4 Bxd4 31. cxd4 Rxg4 32.
  Rxg4 Qxg4 33. Qe3 Kf7 34. Kd2 Qe6 35. Qf4+ Ke7 36. Qc7+ Qd7 37. Qxd7+ Kxd7
  38. Kc3 Kd6 39. Kb4 b5 40. b3 axb3 41. Kxb3 c5 42. dxc5+ Kxc5 43. Kc3 Kc6
  44. Kb4 Kb6 45. a4 bxa4 46. Kxa4 1/2-1/2'''
  
  # If you have the Lichess elite database, yoou can parse the entire thing by uncommenting here
  
  #with open('/lichess_elite_2020-06.pgn') as f:
  #    games = f.read()
  
  games = games.replace("\n\n[", "||||[")
  pgns = games.split("||||")
  
  import time
  
  fens = {}
  start = datetime.datetime.utcnow()
  
  for i, pgn in enumerate(pgns):
      game = chess.pgn.read_game(io.StringIO(pgn)) #parse each pgn string
      if game.errors:
          continue
  
      parent_pos = None
      move_num = 0
      
      while game:
          move_num += 1
          if move_num > 40: #practically, we only need to first 20 moves for our book
              break
  
          game=game.next()
          
          if game:
              board = game.board()
              pos = board.fen(en_passant='fen') #grab fen string of current position
              move = game.san() #what was the last move made?
  
              if pos not in fens:
                  fens[pos]={}
  
              if parent_pos:
                  #in the previous position, the current move was made X times
                  if pos in fens[parent_pos]:
                      fens[parent_pos][move] += 1
                  else:
                      fens[parent_pos][move] = 1
  
          parent_pos=pos
  
  print('Run time: ', (datetime.datetime.utcnow()-start).seconds)
  
  ```
  
  This yields a data structure with FEN strings as keys, and the most common replies as values. E.g. after parsing the 2 games, e4 is followed by g6 once and c5 once.
  
  ```
  {
      'rnbqkbnr/pppppppp/8/8/4P3/8/PPPP1PPP/RNBQKBNR b KQkq e3 0 1': {
          'c5': 1,
          'g6': 1}
  ...
  }```
  
  ```
  
  **Tags:** database, pgn, fen, python
  
  > **Comments:**
  > **username_1**: "Fast" is relative: Object-oriented+Python = you see the taillights of me old FORTRAN guy :-) Still, I program in Python a lot and a simple string parsing app should work fast enough (maybe unless you have millions of games), so mind to give some concrete timing values? And structure and example data for your variables? It might work faster the obvious way: starting from RNBQ.../ and parse the moves of the PGN directly, updating the FEN.  
  > **username_0**: @username_1 One caveat is that I need the strings to comply with the FEN standard for my downstream use case. That means encoding legal en-passant moves, and so on. So, for that reason, I'd love to NOT generate the FEN strings myself!  
  > **username_0**: @username_1 I updated the post with complete code. Note that the script's able to process 100 games every 2 seconds, which means my little database of 400k games will take over 2 hours, and larger databases will take forever.  
  > **username_1**: I took a look at the documentation of PythonChess. First Impression: Very meticulous, very by-the-book-of-good-programming. But your use case needs speed (100 games/2 seconds sounds slow to me either) and it might be more effective to grin and bear it, throw the nice features of PythonChess out of the window, program from scratch and optimize for speed.  
  > **username_2**: FYI, `game.board()` is expensive because it is [re-evaluating the entire game from move one](https://python-chess.readthedocs.io/en/latest/_modules/chess/pgn.html#ChildNode) every time you call that method! Since the logic in that method does not look terribly complicated, you could probably rewrite it yourself to iterate over each board in a single pass.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2021-07-19. Score: 5 | Accepted answer
  
  If you are not tied to using PythonChess and simply want FEN encodings of each position, you might like to consider using my free, open-source [pgn-extract](https://www.cs.kent.ac.uk/%7Edjb/pgn-extract/) program with its -Wepd option to output in EPD format. It is a compiled C program so likely to be significantly faster for this use case. For instance:
  
  ```
  pgn-extract --quiet -Wepd games.pgn
  
  ```
  
  Here are the first few lines of its output for one of the games quoted in the OP:
  
  ```
  rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - c0 sandstorm00-yendorzerep Rated Blitz game 2020.06.01; c1 1/2-1/2;
  rnbqkbnr/pppppppp/8/8/4P3/8/PPPP1PPP/RNBQKBNR b KQkq e3 c0 sandstorm00-yendorzerep Rated Blitz game 2020.06.01; c1 1/2-1/2;
  rnbqkbnr/pppppp1p/6p1/8/4P3/8/PPPP1PPP/RNBQKBNR w KQkq - c0 sandstorm00-yendorzerep Rated Blitz game 2020.06.01; c1 1/2-1/2;
  
  ```
  
  If you wish to suppress irrelevant en passant squares then there is also the --nofauxep option to do that.
  
  > **Comments:**
  > **username_0**: I compiled and tested pgn-extract by piping the first 1000 games in my database. The FENs were generated in about 1 second. So, yes, I think it's worth making the switch! Even considered writing a python-wrapper for pgn-extract, but unsure if there is enough demand to justify that.  
  > **username_3**: I am glad that worked for you.  
  
  ---
  
  ### Answer by username_4 on 2022-09-20. Score: 0
  
  As username_2 indicated, you're reevaluating your entire position after every move. A better way to code would be to use the [`next`](https://python-chess.readthedocs.io/en/latest/pgn.html?highlight=next#chess.pgn.GameNode.next) method. Below is a code snippet that should give you an idea:
  
  ```
  import chess.pgn
  pgn = open("anderssen_kieseritzky_1851.pgn")
  mygame=chess.pgn.read_game(pgn)
  while mygame.next():
      mygame=mygame.next()
      print(mygame.board().fen())
  
  ```
  
  > **Comments:**
  > **username_5**: `AttributeError: 'Game' object has no attribute 'next'`  
  > **username_4**: next is not an attribute but a function. Use next().  
  > **username_4**: @username_5 hope that helps  
  > **username_5**: I think yours is an older version of python-chess. The docs currently recommend iterating over `game.mainline_moves()` https://python-chess.readthedocs.io/en/latest/pgn.html  
  > **username_4**: @username_5 I am using the latest version. Here's the documentation - https://python-chess.readthedocs.io/en/latest/pgn.html#chess.pgn.GameNode.next Can you pls check  
  
  ---
  
  