Answers.Body: <blockquote>
  <p>So how am I supposed to consider hypothetical board positions starting from a given state?</p>
  </blockquote>
  <p>In an environment designed for model-free reinforcement learning, you <em>cannot</em>, at least not via any of the public functions. That's because look-ahead search is not really a part of standard RL toolkit.</p>
  <p>If you want to run a model-based algorithm for tree search algorithms, you need a separate game model implemented. One model instance is maintaining state for the environment, and the second model instance can be used for planning. Sometimes gym libraries do decompose into an environment wrapper and a game engine that you could do this with, but that needs to be checked per environment and there is no standard approach here. The only rule is that the standard gym environment interface has to exist (e.g. <code>reset</code> and <code>step</code> functions).</p>
  <p>A lot of the simpler environments directly implement game rules as functions within a single environment class, and cannot be separated out. This is the case with <code>tictactoe_v3</code> (the source for the <code>step</code> method <a href="https://pettingzoo.farama.org/_modules/pettingzoo/classic/tictactoe/tictactoe/#raw_env.step" rel="nofollow noreferrer">is a long function with significant game rules implemented, not available separately</a>) - in which case the normal option is to implement a <em>new</em> game model and use that for search/planning algorithms.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-24T00:52:11.817Z
Answers.Text: So you need to build a TicTacToe class from scratch in this case?  Why use Gymnasium at all, if you have to build a model of the game from scratch (said game model could trivially implement a step() function, so why use Gymnasium)? Also, will this be the case for Connect Four and Chess?
  
  This seems incredibly inconvenient, I thought Gymnasium was intended to be agnostic to which reinforcement learning approach you were taking.
Answers.UserDisplayName: Ben G
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-24T01:00:45.897Z
Answers.Text: Note that I am attempting to implement the AlphaZero approach to TicTacToe, and later to Connect Four and Chess.  I'm amazed that the most prominent RL environment would not support one of the most prominent approaches to RL in recent history.
Answers.UserDisplayName: Ben G
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-24T11:41:12.157Z
Answers.Text: @BenG it is not as inconvenient as you seem to think. You will still spend far longer on your RL implementation. Either way, I didn't design or write this library. Don't shoot the messenger, and I hope you find the answer useful as it is factual
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-24T11:45:19.560Z
Answers.Text: Gym is not very useful for two player games in any case, as it doesn't really support multiple agent training. When I implemented a Connect 4 agent, I just wrote the model. For chess, maybe look for an engine.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-24T18:03:28.727Z
Answers.Text: The dissapointing thing is that I was looking forward to a standardized environment for playing these games, with a common interface.  So that the same system could play multiple games.  But it seems like no one has created such a common interface and you need to implement each game individually
Answers.UserDisplayName: Ben G
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-24T19:03:53.727Z
Answers.Text: @BenG: There *might* be some library out there that has a few multiplayer games and a standardised interface to make agents for them. But that's not Gym, and not Gym's purpose.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-23T23:36:01.080Z
Answers.LastActivityDate: 2024-02-23T23:36:01.080Z
Answers.UserDisplayName: Neil Slater
Body: <p>I'm attempting to create a Tic Tac Toe player using MCTS.  For the game environment, I'm using <a href="https://pettingzoo.farama.org/environments/classic/tictactoe/" rel="nofollow noreferrer">Tic Tac Toe</a> from the Gymnasium/Petting Zoo environment.</p>
  <p>Running MCTS on Tic Tac Toe requires simulating hypothetical board states.  To do this, my idea is to copy the state of the board at the root node, creating a new environment where I could simulate a hypothetical move.</p>
  <p>I don't see any built in functions for this, so I attempted to see if <code>copy.deepcopy()</code> would copy the board state for me:</p>
  <pre><code>import copy
  from pettingzoo.classic import tictactoe_v3
  
  env = tictactoe_v3.env(render_mode=None)
  env.reset(seed=1)
  
  env = copy.deepcopy(env)
  env.step(0)
  </code></pre>
  <p>This gives an error: <code>AttributeError: 'raw_env' object has no attribute '_cumulative_rewards'</code></p>
  <p>Clearly, deepcopy is not fully copying the board state.  So how am I supposed to consider hypothetical board positions starting from a given state?  Technically I could replay every action in the history to get the new board there, but that seems wildly inefficient, especially since I plan to play more complex games in the gym environment after this.</p>
  
Tags: <reinforcement-learning><monte-carlo-tree-search><gym>
Title: Gymnasium/Petting Zoo: Creating a copy of the board/env
UserDisplayName: Ben G
ThreadText: # Gymnasium/Petting Zoo: Creating a copy of the board/env
  
  (Asked by: username_0 on 2024-02-23)
  
  I'm attempting to create a Tic Tac Toe player using MCTS. For the game environment, I'm using [Tic Tac Toe](https://pettingzoo.farama.org/environments/classic/tictactoe/) from the Gymnasium/Petting Zoo environment.
  
  Running MCTS on Tic Tac Toe requires simulating hypothetical board states. To do this, my idea is to copy the state of the board at the root node, creating a new environment where I could simulate a hypothetical move.
  
  I don't see any built in functions for this, so I attempted to see if `copy.deepcopy()` would copy the board state for me:
  
  ```
  import copy
  from pettingzoo.classic import tictactoe_v3
  
  env = tictactoe_v3.env(render_mode=None)
  env.reset(seed=1)
  
  env = copy.deepcopy(env)
  env.step(0)
  
  ```
  
  This gives an error: `AttributeError: 'raw_env' object has no attribute '_cumulative_rewards'`
  
  Clearly, deepcopy is not fully copying the board state. So how am I supposed to consider hypothetical board positions starting from a given state? Technically I could replay every action in the history to get the new board there, but that seems wildly inefficient, especially since I plan to play more complex games in the gym environment after this.
  
  **Tags:** reinforcement-learning, monte-carlo-tree-search, gym
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-02-23. Score: 1 | Accepted answer
  
  > So how am I supposed to consider hypothetical board positions starting from a given state?
  
  In an environment designed for model-free reinforcement learning, you *cannot*, at least not via any of the public functions. That's because look-ahead search is not really a part of standard RL toolkit.
  
  If you want to run a model-based algorithm for tree search algorithms, you need a separate game model implemented. One model instance is maintaining state for the environment, and the second model instance can be used for planning. Sometimes gym libraries do decompose into an environment wrapper and a game engine that you could do this with, but that needs to be checked per environment and there is no standard approach here. The only rule is that the standard gym environment interface has to exist (e.g. `reset` and `step` functions).
  
  A lot of the simpler environments directly implement game rules as functions within a single environment class, and cannot be separated out. This is the case with `tictactoe_v3` (the source for the `step` method [is a long function with significant game rules implemented, not available separately](https://pettingzoo.farama.org/_modules/pettingzoo/classic/tictactoe/tictactoe/#raw_env.step)) - in which case the normal option is to implement a *new* game model and use that for search/planning algorithms.
  
  > **Comments:**
  > **username_0**: So you need to build a TicTacToe class from scratch in this case?  Why use Gymnasium at all, if you have to build a model of the game from scratch (said game model could trivially implement a step() function, so why use Gymnasium)? Also, will this be the case for Connect Four and Chess?  This seems incredibly inconvenient, I thought Gymnasium was intended to be agnostic to which reinforcement learning approach you were taking.  
  > **username_0**: Note that I am attempting to implement the AlphaZero approach to TicTacToe, and later to Connect Four and Chess.  I'm amazed that the most prominent RL environment would not support one of the most prominent approaches to RL in recent history.  
  > **username_1**: @username_0 it is not as inconvenient as you seem to think. You will still spend far longer on your RL implementation. Either way, I didn't design or write this library. Don't shoot the messenger, and I hope you find the answer useful as it is factual  
  > **username_1**: Gym is not very useful for two player games in any case, as it doesn't really support multiple agent training. When I implemented a Connect 4 agent, I just wrote the model. For chess, maybe look for an engine.  
  > **username_0**: The dissapointing thing is that I was looking forward to a standardized environment for playing these games, with a common interface.  So that the same system could play multiple games.  But it seems like no one has created such a common interface and you need to implement each game individually  
  > **username_1**: @username_0: There *might* be some library out there that has a few multiplayer games and a standardised interface to make agents for them. But that's not Gym, and not Gym's purpose.  
  
  ---
  
  