Answers.Body: <p>So, both the last return values of reset() and step() are the <code>info</code> element, which should be a a dictionary, not a float/int (which I used as a placeholder). Replacing this with a dummy dict fixes the problem.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-16T14:03:08.440Z
Answers.LastActivityDate: 2024-08-16T14:03:08.440Z
Answers.UserDisplayName: DominikS
Body: <p>I want to train my Gymnasium model using stable-baselines3 PPO. I have used the structure below before, however for my particular problem, I keep getting the following problem when calling PPO.train():</p>
  <pre><code>File ~\.conda\envs\reinforce\Lib\site-packages\stable_baselines3\common\vec_env\dummy_vec_env.py:65, in DummyVecEnv.step_wait(self)
   62 self.buf_dones[env_idx] = terminated or truncated
   63 # See https://github.com/openai/gym/issues/3102
   64 # Gym 0.26 introduces a breaking change
   ---&gt; 65 self.buf_infos[env_idx][&quot;TimeLimit.truncated&quot;] = truncated and not terminated
   67 if self.buf_dones[env_idx]:
   68     # save final observation where user can get it, then reset
   69     self.buf_infos[env_idx][&quot;terminal_observation&quot;] = obs
  
   TypeError: 'int' object does not support item assignment
  </code></pre>
  <p>Since it has pretty much the same structure as the classes I have used before, I am surprised why I get an error, I can't find the mistake. Does anyone have more insights?</p>
  <p>The key structure of the my Gym class is as follows:</p>
  <pre><code>class Cells(gym.Env):
  
      def __init__(
          self,
          penalty_factor=10.0,
          action_space=spaces.Box(
              low=np.array([0.0, 0.0, 0.0]),  # minima for delta mu_i
              high=np.array([1.0, 1.0, 1.0]),  # maxima for delta mu_i
              dtype=np.float64,
          ),  # action space
          observation_space=spaces.Dict(
              spaces={
                  &quot;mu_1&quot;: spaces.Box(0.0, 1.0),
                  &quot;mu_2&quot;: spaces.Box(0.0, 1.0),
                  &quot;mu_3&quot;: spaces.Box(0.0, 1.0),
                  &quot;d_1&quot;: spaces.Box(-5.0, 5.0),
                  &quot;d_2&quot;: spaces.Box(-5.0, 5.0),
              }
          ),  # state space
          T_max=100,
      ):
          self.action_space = action_space
          self.observation_space = observation_space
  </code></pre>
  <p>....</p>
  <pre><code>    def get_state(self):
          return {
              &quot;mu_1&quot;: np.array([self.mu_1], dtype=np.float64),
              &quot;mu_2&quot;: np.array([self.mu_2], dtype=np.float64),
              ...
          }
  
      def get_done(self):
          if self.t &gt;= self.T_max:
              return True, False
          else:
              return False, False
  
      def reset(self, **kwargs):
          self.t = 0
          self.mu_1 = 0
          ...
  
          return self.get_state(), 0.0
  </code></pre>
  <p>...</p>
  <pre><code>def step(self, action):
          
          delta_mu_1, delta_mu_2, delta_mu_3 = action
          self.t += self.dt
  
          self.mu_1 += delta_mu_1 * self.dt
          self.mu_2 += delta_mu_2 * self.dt
          self.mu_3 += delta_mu_3 * self.dt
          
          # reward
          reward = self.get_reward()
  
          terminated, truncated = self.get_done()
          if terminated:
              state, _ = self.reset()
          else:
              state = self.get_state()
          return state, reward, terminated, truncated, 0
  </code></pre>
  <p>The first call of step() during training phase completes successfully, and when I print out the return values of step(), I get sensible answers. So I am pretty sure it is not an error in step() itself.</p>
  
Tags: <gym><stable-baselines>
Title: Stable Baselines: 'int' object does not support item assignment
UserDisplayName: DominikS
ThreadText: # Stable Baselines: 'int' object does not support item assignment
  
  (Asked by: username_0 on 2024-08-16)
  
  I want to train my Gymnasium model using stable-baselines3 PPO. I have used the structure below before, however for my particular problem, I keep getting the following problem when calling PPO.train():
  
  ```
  File ~\.conda\envs\reinforce\Lib\site-packages\stable_baselines3\common\vec_env\dummy_vec_env.py:65, in DummyVecEnv.step_wait(self)
   62 self.buf_dones[env_idx] = terminated or truncated
   63 # See https://github.com/openai/gym/issues/3102
   64 # Gym 0.26 introduces a breaking change
   ---> 65 self.buf_infos[env_idx]["TimeLimit.truncated"] = truncated and not terminated
   67 if self.buf_dones[env_idx]:
   68     # save final observation where user can get it, then reset
   69     self.buf_infos[env_idx]["terminal_observation"] = obs
  
   TypeError: 'int' object does not support item assignment
  
  ```
  
  Since it has pretty much the same structure as the classes I have used before, I am surprised why I get an error, I can't find the mistake. Does anyone have more insights?
  
  The key structure of the my Gym class is as follows:
  
  ```
  class Cells(gym.Env):
  
      def __init__(
          self,
          penalty_factor=10.0,
          action_space=spaces.Box(
              low=np.array([0.0, 0.0, 0.0]),  # minima for delta mu_i
              high=np.array([1.0, 1.0, 1.0]),  # maxima for delta mu_i
              dtype=np.float64,
          ),  # action space
          observation_space=spaces.Dict(
              spaces={
                  "mu_1": spaces.Box(0.0, 1.0),
                  "mu_2": spaces.Box(0.0, 1.0),
                  "mu_3": spaces.Box(0.0, 1.0),
                  "d_1": spaces.Box(-5.0, 5.0),
                  "d_2": spaces.Box(-5.0, 5.0),
              }
          ),  # state space
          T_max=100,
      ):
          self.action_space = action_space
          self.observation_space = observation_space
  
  ```
  
  ....
  
  ```
      def get_state(self):
          return {
              "mu_1": np.array([self.mu_1], dtype=np.float64),
              "mu_2": np.array([self.mu_2], dtype=np.float64),
              ...
          }
  
      def get_done(self):
          if self.t >= self.T_max:
              return True, False
          else:
              return False, False
  
      def reset(self, **kwargs):
          self.t = 0
          self.mu_1 = 0
          ...
  
          return self.get_state(), 0.0
  
  ```
  
  ...
  
  ```
  def step(self, action):
          
          delta_mu_1, delta_mu_2, delta_mu_3 = action
          self.t += self.dt
  
          self.mu_1 += delta_mu_1 * self.dt
          self.mu_2 += delta_mu_2 * self.dt
          self.mu_3 += delta_mu_3 * self.dt
          
          # reward
          reward = self.get_reward()
  
          terminated, truncated = self.get_done()
          if terminated:
              state, _ = self.reset()
          else:
              state = self.get_state()
          return state, reward, terminated, truncated, 0
  
  ```
  
  The first call of step() during training phase completes successfully, and when I print out the return values of step(), I get sensible answers. So I am pretty sure it is not an error in step() itself.
  
  **Tags:** gym, stable-baselines
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2024-08-16. Score: 0
  
  So, both the last return values of reset() and step() are the `info` element, which should be a a dictionary, not a float/int (which I used as a placeholder). Replacing this with a dummy dict fixes the problem.
  
  
  ---
  
  