Answers.Body: <blockquote>
  <p>Q1. What is this gym environment warning all about, and how should it be properly addressed?</p>
  </blockquote>
  <p>The development of <code>gym</code> has ended I think. Anyway the error is about an old usage of <code>numpy</code>, so probably you have installed a newer version of it.</p>
  <blockquote>
  <p>Q2. Why are there two environments, gym and gymnasium, that do the same thing?</p>
  </blockquote>
  <p>In principle, they do the same thing but <code>gym</code> was developed by OpenAI and <code>gymnasium</code> by Farama Foundation, which inherited <code>gym</code> basically. You should use <code>gymnasium</code> from now on, since is in actual development. This means that initially <code>gym</code> and <code>gymnasium</code> were almost identical but now their are diverging, even if the underlying concepts are still the same.</p>
  <blockquote>
  <p>Q3. What's the distinction between &quot;terminated&quot; and &quot;truncated&quot;?</p>
  </blockquote>
  <p><code>truncated</code> indicates that the episode ended due to a time limit, i.e. the current timestep have exceeded the time limit so the episode ends. This also indicated that you have to <em>bootstrap</em> the Q value estimate.</p>
  <p><code>terminated</code> instead is when the agent have reached a terminal state, therefore the episode naturally ends. You should use both to understand when an episode terminates, e.g. <code>if truncated or terminated: ...</code>, but the two flags are useful to determine when or not to bootstrap: you don't do that on termination, for example.</p>
  <blockquote>
  <p>Q4. If I want to continue training the agent in stages and test its intermediate performance, how can I do that?</p>
  </blockquote>
  <p>You need to save the Q-table (or, in general, the weights/parameters of a function approximator like a neural-net) but also the current value of the <code>episilon</code> (if using some schedule, so if changing during the episodes) and similar. Then you instantiate the agent, load the various parameters, and continue training where you left.</p>
  <blockquote>
  <p>Q5. Is there a way to get some documentation or help about the environment?</p>
  </blockquote>
  <p>Try <code>help(env)</code>, the official website (<a href="https://gymnasium.farama.org/index.html" rel="nofollow noreferrer">https://gymnasium.farama.org/index.html</a>), or read the env's source code.</p>
  <blockquote>
  <p>Q6. How to use render_mode=&quot;human&quot; correctly?</p>
  </blockquote>
  <p>If you set <code>render_mode=&quot;human&quot;</code> gymnasium will render at each <code>step()</code> and even <code>reset()</code>: this is something that <code>gym</code> not used to do.</p>
  <p>I'm not sure but you may need to wrap the env with this <code>gymnasium.wrappers.RenderCollection</code> (<a href="https://gymnasium.farama.org/api/wrappers/misc_wrappers/#gymnasium.wrappers.RenderCollection" rel="nofollow noreferrer">docs</a>) to trigger the rendering only when <code>render()</code> is called - I haven't tested that, so I can be wrong.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-23T20:40:01.710Z
Answers.Text: Thanks a lot! To summarize,
   / - `gymnasium` environments are the way to go
   / - `help(env)` prints documentation about environment
   / - need to learn about bootstrapping the Q value estimate to use `truncated` flag
   / - to resume training need both Q-table and epsilon value
   / - check `gymnasium.wrappers` for advanced rendering options
Answers.UserDisplayName: Boris L.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-25T08:59:56.737Z
Answers.Text: @BorisL. Yeah, that's all you need to do and know in short. Also bootstrapping is something you'll find in Q-learning and TD-learning. I think there is some good answer about it, in AI SE.
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-22T13:32:11.730Z
Answers.LastActivityDate: 2023-09-22T13:32:11.730Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>As I'm new to the AI/ML field, I'm still learning from various online materials. In this particular instance, I've been studying the <a href="https://youtube.com/playlist?list=PLZbbT5o_s2xoWNVdDudn51XM8lOuZ_Njv&amp;si=Ck2fqwB6CXttv8L9" rel="nofollow noreferrer">Reinforcement Learning tutorial by deeplizard</a>, specifically focusing on videos 8 through 10.</p>
  <p>After attempting to replicate the example that demonstrates how to train an agent in the gym's FrozenLake environment, I encountered some unexpected issues. Although I successfully debugged the code and got the algorithm to work, I can't confidently claim that I fully comprehend if this is the correct approach. Therefore, I have specific questions related to this exercise that I intend to post within this single thread. Please feel free to answer any of the questions at your convenience.</p>
  <h1>Q1</h1>
  <p><strong>What is this <code>gym</code> environment warning all about, and how should it be properly addressed?</strong></p>
  <pre><code>/home/boris/.local/lib/python3.11/site-packages/gym/utils/passive_env_checker.py:233: DeprecationWarning: `np.bool8` is a deprecated alias for `np.bool_`.  (Deprecated NumPy 1.24)
    if not isinstance(terminated, (bool, np.bool8)):
  </code></pre>
  <p>So far, I've discovered two methods to handle this warning: One is to simply  suppress the warning as follows:</p>
  <pre><code>from warnings import filterwarnings
  filterwarnings(action='ignore', category=DeprecationWarning, message='`np.bool8` is a deprecated alias')
  </code></pre>
  <p>Another is to replace the <code>gym</code> environment with the <code>gymnasium</code> environment, which does not produce this warning.</p>
  <pre><code>#import gym
  import gymnasium as gym
  </code></pre>
  <p>This brings me to my second question.</p>
  <h1>Q2</h1>
  <p><strong>Why are there two environments, <code>gym</code> and <code>gymnasium</code>, that do the same thing?</strong>
  Most online examples use <code>gym</code>, but I believe <code>gymnasium</code> is a better choice. Am I mistaken?</p>
  <h1>Q3</h1>
  <p><strong>What's the distinction between &quot;terminated&quot; and &quot;truncated&quot;?</strong> I've made numerous attempts, but I couldn't find a situation where they diverge. They consistently evaluate to either both True or both False.</p>
  <p>In the current 'gym' version, I have to use &quot;FrozenLake-v1&quot; (originally, &quot;FrozenLake-v0&quot; was used in the tutorial), and this modification causes the code to break because the environment now expects 5 values instead of 4. Initially, I fixed this issue as follows:</p>
  <pre><code>new_state, reward, done, UNKNOWN, info = env.step(action)
  </code></pre>
  <p>I discovered later that the correct name for this variable is &quot;truncated.&quot; Consequently, I adjusted the code as follows:</p>
  <pre><code>new_state, reward, terminated, truncated, info = env.step(action)
  done = terminated or truncated
  </code></pre>
  <p>Nonetheless, I remain perplexed regarding the usage of &quot;terminated&quot; and &quot;truncated.&quot; While experimenting with various algorithm options, such as <code>max_steps_per_episode</code>, I was unable to get them to differ. In all cases, they either both evaluate to True or both to False.</p>
  <h1>Q4</h1>
  <p><strong>If I want to continue training the agent in stages and test its intermediate performance, how can I do that?</strong> I was thinking it's just a matter of saving the q_table, then loading the saved q_table and replacing the current q_table initialization with the new one before resuming training. Or is it not that simple?</p>
  <pre><code>saved_q_table = anydbm.open(filename, 'r')
  
  #q_table = np.zeros((state_space_size, action_space_size))
  q_table = pickle.loads(saved_q_table)
  </code></pre>
  <h1>Q5</h1>
  <p><strong>Is there a way to get some documentation or help about the environment?</strong></p>
  <p>I'm seeking a resource that I can print and reference while coding to gain a better understanding of the environment's parameters. For instance, in scikit-learn, you can achieve this with the following code:</p>
  <pre><code>from sklearn.neighbors import KNeighborsRegressor
  from pprint import pprint
  pprint(KNeighborsRegressor().get_params())
  </code></pre>
  <p>Resulting in following output:</p>
  <pre><code>{'algorithm': 'auto',
   'leaf_size': 30,
   'metric': 'minkowski',
   'metric_params': None,
   'n_jobs': None,
   'n_neighbors': 5,
   'p': 2,
   'weights': 'uniform'}
  </code></pre>
  <p>It would be great if one could get relevant agent/states parameters from the <code>gym</code> environments in a similar way.</p>
  <h1>Q6</h1>
  <p><strong>How to use <code>render_mode=&quot;human&quot;</code> correctly?</strong> Currently, I'm using <code>render_mode=&quot;ansi&quot;</code> and rendering the environment as follows:</p>
  <pre><code>env = gym.make(&quot;FrozenLake-v1&quot;, map_name=&quot;8x8&quot;, is_slippery=True, render_mode=&quot;ansi&quot;, desc=None)
  
  print(&quot;\033c&quot;)
  env.render()
  print(env.render())
  </code></pre>
  <p>However, when I switch to <code>render_mode=&quot;human&quot;</code>, the environment automatically displays without the need for env.render(). Consequently, the environment renders during training as well, leading to extremely slow training. How can I disable this automatic rendering when using render_mode=&quot;human&quot;?</p>
  <h1>Q7</h1>
  <p><strong>Why doesn't this algorithm work for an 8x8 map size?</strong></p>
  <pre><code>env = gym.make(&quot;FrozenLake-v1&quot;, map_name=&quot;8x8&quot;, is_slippery=True, render_mode=&quot;ansi&quot;, desc=None)
  </code></pre>
  <p>It appears that the agent is being trained, but it fails to learn anything.</p>
  <pre><code>*** Average reward per thousand episodes ***
  
  1000 :  0.0
  2000 :  0.0
  3000 :  0.0
  4000 :  0.0
  5000 :  0.0
  6000 :  0.0
  7000 :  0.0
  8000 :  0.0
  9000 :  0.0
  10000 :  0.0
  </code></pre>
  <p>I can't seem to figure out what needs to be modified. The <code>q_table</code> is the correct size:</p>
  <pre><code>q_table = np.zeros((state_space_size, action_space_size))
  </code></pre>
  <p>I've also increased <code>max_steps_per_episode</code> to 3000. Is it possible that <code>num_episodes = 10000</code> isn't sufficient?</p>
  <h1>By the way, here is the full code I have so far:</h1>
  <pre><code>#!/usr/bin/python
  
  # -*- coding: utf-8 -*-
  
  import numpy as np
  #import gym
  import gymnasium as gym
  import random
  import time
  from warnings import filterwarnings
  
  #filterwarnings(action='ignore', category=DeprecationWarning, message='`np.bool8` is a deprecated alias')
  
  env = gym.make(&quot;FrozenLake-v1&quot;, map_name=&quot;4x4&quot;, is_slippery=True, render_mode=&quot;ansi&quot;, desc=None)
  #env = gym.make(&quot;FrozenLake-v1&quot;, map_name=&quot;4x4&quot;, is_slippery=True, render_mode=&quot;human&quot;, desc=None)
  
  action_space_size = env.action_space.n
  state_space_size = env.observation_space.n
  #print(action_space_size)
  #print(state_space_size)
  
  q_table = np.zeros((state_space_size, action_space_size))
  
  num_episodes = 10000
  max_steps_per_episode = 3000
  
  learning_rate = 0.11
  discount_rate = 0.99
  
  exploration_rate = 1
  max_exploration_rate = 0.99
  min_exploration_rate = 0.01
  exploration_decay_rate = 0.002
  
  rewards_all_episodes = []
  
  # Q-Learnig Algorithm
  for episode in range(num_episodes):
      print(&quot;*** Episode &quot;,  episode+1, &quot; ***&quot;, end=&quot;\r&quot;)
      state = env.reset()[0]
  
      done = False
      rewards_current_episode = 0
  
      for step in range(max_steps_per_episode):
  
          # Exploration-Exploitation Trade-off
          exploration_rate_threshold = random.uniform(0, 1)
          if exploration_rate_threshold &gt; exploration_rate:
              action = np.argmax(q_table[state, :])
          else:
              action = env.action_space.sample()
          new_state, reward, terminated, truncated, info = env.step(action)
          done = terminated or truncated
  
          # Update Q-Table for Q(s,a)
          q_table[state, action] = q_table[state, action] * (1 - learning_rate) + learning_rate * (reward + discount_rate * np.max(q_table[new_state, :]))
  
          state = new_state
          rewards_current_episode += reward
  
          if done == True:
              break
  
      # Exploration rate decay
      exploration_rate = min_exploration_rate + (max_exploration_rate - min_exploration_rate) * np.exp(-exploration_decay_rate * episode)
  
      rewards_all_episodes.append(rewards_current_episode)
  
  # Calsulate and print the average reward per thousand episodes
  rewards_per_thousand_episodes = np.split(np.array(rewards_all_episodes), num_episodes/1000)
  count = 1000
  print(&quot;*** Average reward per thousand episodes ***\n&quot;)
  for r in rewards_per_thousand_episodes:
      print(count, &quot;: &quot;, str(sum(r/1000)))
      count += 1000
  
  # Print updated Q-Table
  print(&quot;\n\n*** Updated Q-Table ***\n&quot;)
  print(q_table)
  input(&quot;Press Enter to continue...&quot;)
  
  num_episodes = 5
  max_steps_per_episode = 99
  
  for episode in range(num_episodes):
      state = env.reset()[0]
  
      done = False
  
      for step in range(max_steps_per_episode):
          print(&quot;\033c&quot;)
          print(&quot;*** Episode &quot;,  episode+1, &quot; ***\n&quot;, step)
          env.render()
          print(env.render())
          time.sleep(0.1)
  
          action = np.argmax(q_table[state, :])
          new_state, reward, done, unknown, info = env.step(action)
  
          # Update Q-Table for Q(s,a)
          q_table[state, action] = q_table[state, action] * (1 - learning_rate) + learning_rate * (reward + discount_rate * np.max(q_table[new_state, :]))
  
          state = new_state
          rewards_current_episode += reward
  
          if done == True:
              print(&quot;\033c&quot;)
              print(&quot;*** Episode &quot;,  episode+1, &quot; ***\n&quot;)
              env.render()
              print(env.render())
              if reward == 1:
                  print(&quot;*** You reached the goal! ***&quot;)
                  time.sleep(3)
              else:
                  print(&quot;*** You fell through a hole! ***&quot;)
                  time.sleep(3)
              #clear_output(wait=True)
              break
  
          if step == max_steps_per_episode - 1:
              print(&quot;*** You failed to complete the task! ***&quot;)
              time.sleep(3)
  </code></pre>
  
Tags: <reinforcement-learning><q-learning><gym><frozen-lake>
Title: My first experience with gym environment has raised many questions, and I need some guidance
UserDisplayName: Boris L.
ThreadText: # My first experience with gym environment has raised many questions, and I need some guidance
  
  (Asked by: username_0 on 2023-09-22)
  
  As I'm new to the AI/ML field, I'm still learning from various online materials. In this particular instance, I've been studying the [Reinforcement Learning tutorial by deeplizard](https://youtube.com/playlist?list=PLZbbT5o_s2xoWNVdDudn51XM8lOuZ_Njv&si=Ck2fqwB6CXttv8L9), specifically focusing on videos 8 through 10.
  
  After attempting to replicate the example that demonstrates how to train an agent in the gym's FrozenLake environment, I encountered some unexpected issues. Although I successfully debugged the code and got the algorithm to work, I can't confidently claim that I fully comprehend if this is the correct approach. Therefore, I have specific questions related to this exercise that I intend to post within this single thread. Please feel free to answer any of the questions at your convenience.
  
  # Q1
  
  **What is this `gym` environment warning all about, and how should it be properly addressed?**
  
  ```
  /home/username_0/.local/lib/python3.11/site-packages/gym/utils/passive_env_checker.py:233: DeprecationWarning: `np.bool8` is a deprecated alias for `np.bool_`.  (Deprecated NumPy 1.24)
    if not isinstance(terminated, (bool, np.bool8)):
  
  ```
  
  So far, I've discovered two methods to handle this warning: One is to simply suppress the warning as follows:
  
  ```
  from warnings import filterwarnings
  filterwarnings(action='ignore', category=DeprecationWarning, message='`np.bool8` is a deprecated alias')
  
  ```
  
  Another is to replace the `gym` environment with the `gymnasium` environment, which does not produce this warning.
  
  ```
  #import gym
  import gymnasium as gym
  
  ```
  
  This brings me to my second question.
  
  # Q2
  
  **Why are there two environments, `gym` and `gymnasium`, that do the same thing?**
  Most online examples use `gym`, but I believe `gymnasium` is a better choice. Am I mistaken?
  
  # Q3
  
  **What's the distinction between "terminated" and "truncated"?** I've made numerous attempts, but I couldn't find a situation where they diverge. They consistently evaluate to either both True or both False.
  
  In the current 'gym' version, I have to use "FrozenLake-v1" (originally, "FrozenLake-v0" was used in the tutorial), and this modification causes the code to break because the environment now expects 5 values instead of 4. Initially, I fixed this issue as follows:
  
  ```
  new_state, reward, done, UNKNOWN, info = env.step(action)
  
  ```
  
  I discovered later that the correct name for this variable is "truncated." Consequently, I adjusted the code as follows:
  
  ```
  new_state, reward, terminated, truncated, info = env.step(action)
  done = terminated or truncated
  
  ```
  
  Nonetheless, I remain perplexed regarding the usage of "terminated" and "truncated." While experimenting with various algorithm options, such as `max_steps_per_episode`, I was unable to get them to differ. In all cases, they either both evaluate to True or both to False.
  
  # Q4
  
  **If I want to continue training the agent in stages and test its intermediate performance, how can I do that?** I was thinking it's just a matter of saving the q\_table, then loading the saved q\_table and replacing the current q\_table initialization with the new one before resuming training. Or is it not that simple?
  
  ```
  saved_q_table = anydbm.open(filename, 'r')
  
  #q_table = np.zeros((state_space_size, action_space_size))
  q_table = pickle.loads(saved_q_table)
  
  ```
  
  # Q5
  
  **Is there a way to get some documentation or help about the environment?**
  
  I'm seeking a resource that I can print and reference while coding to gain a better understanding of the environment's parameters. For instance, in scikit-learn, you can achieve this with the following code:
  
  ```
  from sklearn.neighbors import KNeighborsRegressor
  from pprint import pprint
  pprint(KNeighborsRegressor().get_params())
  
  ```
  
  Resulting in following output:
  
  ```
  {'algorithm': 'auto',
   'leaf_size': 30,
   'metric': 'minkowski',
   'metric_params': None,
   'n_jobs': None,
   'n_neighbors': 5,
   'p': 2,
   'weights': 'uniform'}
  
  ```
  
  It would be great if one could get relevant agent/states parameters from the `gym` environments in a similar way.
  
  # Q6
  
  **How to use `render_mode="human"` correctly?** Currently, I'm using `render_mode="ansi"` and rendering the environment as follows:
  
  ```
  env = gym.make("FrozenLake-v1", map_name="8x8", is_slippery=True, render_mode="ansi", desc=None)
  
  print("\033c")
  env.render()
  print(env.render())
  
  ```
  
  However, when I switch to `render_mode="human"`, the environment automatically displays without the need for env.render(). Consequently, the environment renders during training as well, leading to extremely slow training. How can I disable this automatic rendering when using render\_mode="human"?
  
  # Q7
  
  **Why doesn't this algorithm work for an 8x8 map size?**
  
  ```
  env = gym.make("FrozenLake-v1", map_name="8x8", is_slippery=True, render_mode="ansi", desc=None)
  
  ```
  
  It appears that the agent is being trained, but it fails to learn anything.
  
  ```
  *** Average reward per thousand episodes ***
  
  1000 :  0.0
  2000 :  0.0
  3000 :  0.0
  4000 :  0.0
  5000 :  0.0
  6000 :  0.0
  7000 :  0.0
  8000 :  0.0
  9000 :  0.0
  10000 :  0.0
  
  ```
  
  I can't seem to figure out what needs to be modified. The `q_table` is the correct size:
  
  ```
  q_table = np.zeros((state_space_size, action_space_size))
  
  ```
  
  I've also increased `max_steps_per_episode` to 3000. Is it possible that `num_episodes = 10000` isn't sufficient?
  
  # By the way, here is the full code I have so far:
  
  ```
  #!/usr/bin/python
  
  # -*- coding: utf-8 -*-
  
  import numpy as np
  #import gym
  import gymnasium as gym
  import random
  import time
  from warnings import filterwarnings
  
  #filterwarnings(action='ignore', category=DeprecationWarning, message='`np.bool8` is a deprecated alias')
  
  env = gym.make("FrozenLake-v1", map_name="4x4", is_slippery=True, render_mode="ansi", desc=None)
  #env = gym.make("FrozenLake-v1", map_name="4x4", is_slippery=True, render_mode="human", desc=None)
  
  action_space_size = env.action_space.n
  state_space_size = env.observation_space.n
  #print(action_space_size)
  #print(state_space_size)
  
  q_table = np.zeros((state_space_size, action_space_size))
  
  num_episodes = 10000
  max_steps_per_episode = 3000
  
  learning_rate = 0.11
  discount_rate = 0.99
  
  exploration_rate = 1
  max_exploration_rate = 0.99
  min_exploration_rate = 0.01
  exploration_decay_rate = 0.002
  
  rewards_all_episodes = []
  
  # Q-Learnig Algorithm
  for episode in range(num_episodes):
      print("*** Episode ",  episode+1, " ***", end="\r")
      state = env.reset()[0]
  
      done = False
      rewards_current_episode = 0
  
      for step in range(max_steps_per_episode):
  
          # Exploration-Exploitation Trade-off
          exploration_rate_threshold = random.uniform(0, 1)
          if exploration_rate_threshold > exploration_rate:
              action = np.argmax(q_table[state, :])
          else:
              action = env.action_space.sample()
          new_state, reward, terminated, truncated, info = env.step(action)
          done = terminated or truncated
  
          # Update Q-Table for Q(s,a)
          q_table[state, action] = q_table[state, action] * (1 - learning_rate) + learning_rate * (reward + discount_rate * np.max(q_table[new_state, :]))
  
          state = new_state
          rewards_current_episode += reward
  
          if done == True:
              break
  
      # Exploration rate decay
      exploration_rate = min_exploration_rate + (max_exploration_rate - min_exploration_rate) * np.exp(-exploration_decay_rate * episode)
  
      rewards_all_episodes.append(rewards_current_episode)
  
  # Calsulate and print the average reward per thousand episodes
  rewards_per_thousand_episodes = np.split(np.array(rewards_all_episodes), num_episodes/1000)
  count = 1000
  print("*** Average reward per thousand episodes ***\n")
  for r in rewards_per_thousand_episodes:
      print(count, ": ", str(sum(r/1000)))
      count += 1000
  
  # Print updated Q-Table
  print("\n\n*** Updated Q-Table ***\n")
  print(q_table)
  input("Press Enter to continue...")
  
  num_episodes = 5
  max_steps_per_episode = 99
  
  for episode in range(num_episodes):
      state = env.reset()[0]
  
      done = False
  
      for step in range(max_steps_per_episode):
          print("\033c")
          print("*** Episode ",  episode+1, " ***\n", step)
          env.render()
          print(env.render())
          time.sleep(0.1)
  
          action = np.argmax(q_table[state, :])
          new_state, reward, done, unknown, info = env.step(action)
  
          # Update Q-Table for Q(s,a)
          q_table[state, action] = q_table[state, action] * (1 - learning_rate) + learning_rate * (reward + discount_rate * np.max(q_table[new_state, :]))
  
          state = new_state
          rewards_current_episode += reward
  
          if done == True:
              print("\033c")
              print("*** Episode ",  episode+1, " ***\n")
              env.render()
              print(env.render())
              if reward == 1:
                  print("*** You reached the goal! ***")
                  time.sleep(3)
              else:
                  print("*** You fell through a hole! ***")
                  time.sleep(3)
              #clear_output(wait=True)
              break
  
          if step == max_steps_per_episode - 1:
              print("*** You failed to complete the task! ***")
              time.sleep(3)
  
  ```
  
  **Tags:** reinforcement-learning, q-learning, gym, frozen-lake
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-09-22. Score: 2
  
  > Q1. What is this gym environment warning all about, and how should it be properly addressed?
  
  The development of `gym` has ended I think. Anyway the error is about an old usage of `numpy`, so probably you have installed a newer version of it.
  
  > Q2. Why are there two environments, gym and gymnasium, that do the same thing?
  
  In principle, they do the same thing but `gym` was developed by OpenAI and `gymnasium` by Farama Foundation, which inherited `gym` basically. You should use `gymnasium` from now on, since is in actual development. This means that initially `gym` and `gymnasium` were almost identical but now their are diverging, even if the underlying concepts are still the same.
  
  > Q3. What's the distinction between "terminated" and "truncated"?
  
  `truncated` indicates that the episode ended due to a time limit, i.e. the current timestep have exceeded the time limit so the episode ends. This also indicated that you have to *bootstrap* the Q value estimate.
  
  `terminated` instead is when the agent have reached a terminal state, therefore the episode naturally ends. You should use both to understand when an episode terminates, e.g. `if truncated or terminated: ...`, but the two flags are useful to determine when or not to bootstrap: you don't do that on termination, for example.
  
  > Q4. If I want to continue training the agent in stages and test its intermediate performance, how can I do that?
  
  You need to save the Q-table (or, in general, the weights/parameters of a function approximator like a neural-net) but also the current value of the `episilon` (if using some schedule, so if changing during the episodes) and similar. Then you instantiate the agent, load the various parameters, and continue training where you left.
  
  > Q5. Is there a way to get some documentation or help about the environment?
  
  Try `help(env)`, the official website (<https://gymnasium.farama.org/index.html>), or read the env's source code.
  
  > Q6. How to use render\_mode="human" correctly?
  
  If you set `render_mode="human"` gymnasium will render at each `step()` and even `reset()`: this is something that `gym` not used to do.
  
  I'm not sure but you may need to wrap the env with this `gymnasium.wrappers.RenderCollection` ([docs](https://gymnasium.farama.org/api/wrappers/misc_wrappers/#gymnasium.wrappers.RenderCollection)) to trigger the rendering only when `render()` is called - I haven't tested that, so I can be wrong.
  
  > **Comments:**
  > **username_0**: Thanks a lot! To summarize,  / - `gymnasium` environments are the way to go  / - `help(env)` prints documentation about environment  / - need to learn about bootstrapping the Q value estimate to use `truncated` flag  / - to resume training need both Q-table and epsilon value  / - check `gymnasium.wrappers` for advanced rendering options  
  > **username_1**: @BorisL. Yeah, that's all you need to do and know in short. Also bootstrapping is something you'll find in Q-learning and TD-learning. I think there is some good answer about it, in AI SE.  
  
  ---
  
  