Answers.Body: <blockquote>
  <p>Q1. What is this gym environment warning all about, and how should it be properly addressed?</p>
  </blockquote>
  <p>The development of <code>gym</code> has ended I think. Anyway the error is about an old usage of <code>numpy</code>, so probably you have installed a newer version of it.</p>
  <blockquote>
  <p>Q2. Why are there two environments, gym and gymnasium, that do the same thing?</p>
  </blockquote>
  <p>In principle, they do the same thing but <code>gym</code> was developed by OpenAI and <code>gymnasium</code> by Farama Foundation, which inherited <code>gym</code> basically. You should use <code>gymnasium</code> from now on, since is in actual development. This means that initially <code>gym</code> and <code>gymnasium</code> were almost identical but now their are diverging, even if the underlying concepts are still the same.</p>
  <blockquote>
  <p>Q3. What's the distinction between &quot;terminated&quot; and &quot;truncated&quot;?</p>
  </blockquote>
  <p><code>truncated</code> indicates that the episode ended due to a time limit, i.e. the current timestep have exceeded the time limit so the episode ends. This also indicated that you have to <em>bootstrap</em> the Q value estimate.</p>
  <p><code>terminated</code> instead is when the agent have reached a terminal state, therefore the episode naturally ends. You should use both to understand when an episode terminates, e.g. <code>if truncated or terminated: ...</code>, but the two flags are useful to determine when or not to bootstrap: you don't do that on termination, for example.</p>
  <blockquote>
  <p>Q4. If I want to continue training the agent in stages and test its intermediate performance, how can I do that?</p>
  </blockquote>
  <p>You need to save the Q-table (or, in general, the weights/parameters of a function approximator like a neural-net) but also the current value of the <code>episilon</code> (if using some schedule, so if changing during the episodes) and similar. Then you instantiate the agent, load the various parameters, and continue training where you left.</p>
  <blockquote>
  <p>Q5. Is there a way to get some documentation or help about the environment?</p>
  </blockquote>
  <p>Try <code>help(env)</code>, the official website (<a href="https://gymnasium.farama.org/index.html" rel="nofollow noreferrer">https://gymnasium.farama.org/index.html</a>), or read the env's source code.</p>
  <blockquote>
  <p>Q6. How to use render_mode=&quot;human&quot; correctly?</p>
  </blockquote>
  <p>If you set <code>render_mode=&quot;human&quot;</code> gymnasium will render at each <code>step()</code> and even <code>reset()</code>: this is something that <code>gym</code> not used to do.</p>
  <p>I'm not sure but you may need to wrap the env with this <code>gymnasium.wrappers.RenderCollection</code> (<a href="https://gymnasium.farama.org/api/wrappers/misc_wrappers/#gymnasium.wrappers.RenderCollection" rel="nofollow noreferrer">docs</a>) to trigger the rendering only when <code>render()</code> is called - I haven't tested that, so I can be wrong.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-23T20:40:01.710Z
Answers.Text: Thanks a lot! To summarize,
   / - `gymnasium` environments are the way to go
   / - `help(env)` prints documentation about environment
   / - need to learn about bootstrapping the Q value estimate to use `truncated` flag
   / - to resume training need both Q-table and epsilon value
   / - check `gymnasium.wrappers` for advanced rendering options
Answers.UserDisplayName: Boris L.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-25T08:59:56.737Z
Answers.Text: @BorisL. Yeah, that's all you need to do and know in short. Also bootstrapping is something you'll find in Q-learning and TD-learning. I think there is some good answer about it, in AI SE.
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-22T13:32:11.730Z
Answers.LastActivityDate: 2023-09-22T13:32:11.730Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>As I'm new to the AI/ML field, I'm still learning from various online materials. In this particular instance, I've been studying the <a href="https://youtube.com/playlist?list=PLZbbT5o_s2xoWNVdDudn51XM8lOuZ_Njv&amp;si=Ck2fqwB6CXttv8L9" rel="nofollow noreferrer">Reinforcement Learning tutorial by deeplizard</a>, specifically focusing on videos 8 through 10.</p>
  <p>After attempting to replicate the example that demonstrates how to train an agent in the gym's FrozenLake environment, I encountered some unexpected issues. Although I successfully debugged the code and got the algorithm to work, I can't confidently claim that I fully comprehend if this is the correct approach. Therefore, I have specific questions related to this exercise that I intend to post within this single thread. Please feel free to answer any of the questions at your convenience.</p>
  <h1>Q1</h1>
  <p><strong>What is this <code>gym</code> environment warning all about, and how should it be properly addressed?</strong></p>
  <pre><code>/home/boris/.local/lib/python3.11/site-packages/gym/utils/passive_env_checker.py:233: DeprecationWarning: `np.bool8` is a deprecated alias for `np.bool_`.  (Deprecated NumPy 1.24)
    if not isinstance(terminated, (bool, np.bool8)):
  </code></pre>
  <p>So far, I've discovered two methods to handle this warning: One is to simply  suppress the warning as follows:</p>
  <pre><code>from warnings import filterwarnings
  filterwarnings(action='ignore', category=DeprecationWarning, message='`np.bool8` is a deprecated alias')
  </code></pre>
  <p>Another is to replace the <code>gym</code> environment with the <code>gymnasium</code> environment, which does not produce this warning.</p>
  <pre><code>#import gym
  import gymnasium as gym
  </code></pre>
  <p>This brings me to my second question.</p>
  <h1>Q2</h1>
  <p><strong>Why are there two environments, <code>gym</code> and <code>gymnasium</code>, that do the same thing?</strong>
  Most online examples use <code>gym</code>, but I believe <code>gymnasium</code> is a better choice. Am I mistaken?</p>
  <h1>Q3</h1>
  <p><strong>What's the distinction between &quot;terminated&quot; and &quot;truncated&quot;?</strong> I've made numerous attempts, but I couldn't find a situation where they diverge. They consistently evaluate to either both True or both False.</p>
  <p>In the current 'gym' version, I have to use &quot;FrozenLake-v1&quot; (originally, &quot;FrozenLake-v0&quot; was used in the tutorial), and this modification causes the code to break because the environment now expects 5 values instead of 4. Initially, I fixed this issue as follows:</p>
  <pre><code>new_state, reward, done, UNKNOWN, info = env.step(action)
  </code></pre>
  <p>I discovered later that the correct name for this variable is &quot;truncated.&quot; Consequently, I adjusted the code as follows:</p>
  <pre><code>new_state, reward, terminated, truncated, info = env.step(action)
  done = terminated or truncated
  </code></pre>
  <p>Nonetheless, I remain perplexed regarding the usage of &quot;terminated&quot; and &quot;truncated.&quot; While experimenting with various algorithm options, such as <code>max_steps_per_episode</code>, I was unable to get them to differ. In all cases, they either both evaluate to True or both to False.</p>
  <h1>Q4</h1>
  <p><strong>If I want to continue training the agent in stages and test its intermediate performance, how can I do that?</strong> I was thinking it's just a matter of saving the q_table, then loading the saved q_table and replacing the current q_table initialization with the new one before resuming training. Or is it not that simple?</p>
  <pre><code>saved_q_table = anydbm.open(filename, 'r')
  
  #q_table = np.zeros((state_space_size, action_space_size))
  q_table = pickle.loads(saved_q_table)
  </code></pre>
  <h1>Q5</h1>
  <p><strong>Is there a way to get some documentation or help about the environment?</strong></p>
  <p>I'm seeking a resource that I can print and reference while coding to gain a better understanding of the environment's parameters. For instance, in scikit-learn, you can achieve this with the following code:</p>
  <pre><code>from sklearn.neighbors import KNeighborsRegressor
  from pprint import pprint
  pprint(KNeighborsRegressor().get_params())
  </code></pre>
  <p>Resulting in following output:</p>
  <pre><code>{'algorithm': 'auto',
   'leaf_size': 30,
   'metric': 'minkowski',
   'metric_params': None,
   'n_jobs': None,
   'n_neighbors': 5,
   'p': 2,
   'weights': 'uniform'}
  </code></pre>
  <p>It would be great if one could get relevant agent/states parameters from the <code>gym</code> environments in a similar way.</p>
  <h1>Q6</h1>
  <p><strong>How to use <code>render_mode=&quot;human&quot;</code> correctly?</strong> Currently, I'm using <code>render_mode=&quot;ansi&quot;</code> and rendering the environment as follows:</p>
  <pre><code>env = gym.make(&quot;FrozenLake-v1&quot;, map_name=&quot;8x8&quot;, is_slippery=True, render_mode=&quot;ansi&quot;, desc=None)
  
  print(&quot;\033c&quot;)
  env.render()
  print(env.render())
  </code></pre>
  <p>However, when I switch to <code>render_mode=&quot;human&quot;</code>, the environment automatically displays without the need for env.render(). Consequently, the environment renders during training as well, leading to extremely slow training. How can I disable this automatic rendering when using render_mode=&quot;human&quot;?</p>
  <h1>Q7</h1>
  <p><strong>Why doesn't this algorithm work for an 8x8 map size?</strong></p>
  <pre><code>env = gym.make(&quot;FrozenLake-v1&quot;, map_name=&quot;8x8&quot;, is_slippery=True, render_mode=&quot;ansi&quot;, desc=None)
  </code></pre>
  <p>It appears that the agent is being trained, but it fails to learn anything.</p>
  <pre><code>*** Average reward per thousand episodes ***
  
  1000 :  0.0
  2000 :  0.0
  3000 :  0.0
  4000 :  0.0
  5000 :  0.0
  6000 :  0.0
  7000 :  0.0
  8000 :  0.0
  9000 :  0.0
  10000 :  0.0
  </code></pre>
  <p>I can't seem to figure out what needs to be modified. The <code>q_table</code> is the correct size:</p>
  <pre><code>q_table = np.zeros((state_space_size, action_space_size))
  </code></pre>
  <p>I've also increased <code>max_steps_per_episode</code> to 3000. Is it possible that <code>num_episodes = 10000</code> isn't sufficient?</p>
  <h1>By the way, here is the full code I have so far:</h1>
  <pre><code>#!/usr/bin/python
  
  # -*- coding: utf-8 -*-
  
  import numpy as np
  #import gym
  import gymnasium as gym
  import random
  import time
  from warnings import filterwarnings
  
  #filterwarnings(action='ignore', category=DeprecationWarning, message='`np.bool8` is a deprecated alias')
  
  env = gym.make(&quot;FrozenLake-v1&quot;, map_name=&quot;4x4&quot;, is_slippery=True, render_mode=&quot;ansi&quot;, desc=None)
  #env = gym.make(&quot;FrozenLake-v1&quot;, map_name=&quot;4x4&quot;, is_slippery=True, render_mode=&quot;human&quot;, desc=None)
  
  action_space_size = env.action_space.n
  state_space_size = env.observation_space.n
  #print(action_space_size)
  #print(state_space_size)
  
  q_table = np.zeros((state_space_size, action_space_size))
  
  num_episodes = 10000
  max_steps_per_episode = 3000
  
  learning_rate = 0.11
  discount_rate = 0.99
  
  exploration_rate = 1
  max_exploration_rate = 0.99
  min_exploration_rate = 0.01
  exploration_decay_rate = 0.002
  
  rewards_all_episodes = []
  
  # Q-Learnig Algorithm
  for episode in range(num_episodes):
      print(&quot;*** Episode &quot;,  episode+1, &quot; ***&quot;, end=&quot;\r&quot;)
      state = env.reset()[0]
  
      done = False
      rewards_current_episode = 0
  
      for step in range(max_steps_per_episode):
  
          # Exploration-Exploitation Trade-off
          exploration_rate_threshold = random.uniform(0, 1)
          if exploration_rate_threshold &gt; exploration_rate:
              action = np.argmax(q_table[state, :])
          else:
              action = env.action_space.sample()
          new_state, reward, terminated, truncated, info = env.step(action)
          done = terminated or truncated
  
          # Update Q-Table for Q(s,a)
          q_table[state, action] = q_table[state, action] * (1 - learning_rate) + learning_rate * (reward + discount_rate * np.max(q_table[new_state, :]))
  
          state = new_state
          rewards_current_episode += reward
  
          if done == True:
              break
  
      # Exploration rate decay
      exploration_rate = min_exploration_rate + (max_exploration_rate - min_exploration_rate) * np.exp(-exploration_decay_rate * episode)
  
      rewards_all_episodes.append(rewards_current_episode)
  
  # Calsulate and print the average reward per thousand episodes
  rewards_per_thousand_episodes = np.split(np.array(rewards_all_episodes), num_episodes/1000)
  count = 1000
  print(&quot;*** Average reward per thousand episodes ***\n&quot;)
  for r in rewards_per_thousand_episodes:
      print(count, &quot;: &quot;, str(sum(r/1000)))
      count += 1000
  
  # Print updated Q-Table
  print(&quot;\n\n*** Updated Q-Table ***\n&quot;)
  print(q_table)
  input(&quot;Press Enter to continue...&quot;)
  
  num_episodes = 5
  max_steps_per_episode = 99
  
  for episode in range(num_episodes):
      state = env.reset()[0]
  
      done = False
  
      for step in range(max_steps_per_episode):
          print(&quot;\033c&quot;)
          print(&quot;*** Episode &quot;,  episode+1, &quot; ***\n&quot;, step)
          env.render()
          print(env.render())
          time.sleep(0.1)
  
          action = np.argmax(q_table[state, :])
          new_state, reward, done, unknown, info = env.step(action)
  
          # Update Q-Table for Q(s,a)
          q_table[state, action] = q_table[state, action] * (1 - learning_rate) + learning_rate * (reward + discount_rate * np.max(q_table[new_state, :]))
  
          state = new_state
          rewards_current_episode += reward
  
          if done == True:
              print(&quot;\033c&quot;)
              print(&quot;*** Episode &quot;,  episode+1, &quot; ***\n&quot;)
              env.render()
              print(env.render())
              if reward == 1:
                  print(&quot;*** You reached the goal! ***&quot;)
                  time.sleep(3)
              else:
                  print(&quot;*** You fell through a hole! ***&quot;)
                  time.sleep(3)
              #clear_output(wait=True)
              break
  
          if step == max_steps_per_episode - 1:
              print(&quot;*** You failed to complete the task! ***&quot;)
              time.sleep(3)
  </code></pre>
  
Tags: <reinforcement-learning><q-learning><gym><frozen-lake>
Title: My first experience with gym environment has raised many questions, and I need some guidance
UserDisplayName: Boris L.
ThreadText: # My first experience with gym environment has raised many questions, and I need some guidance
  
  (Asked by: username_0 on 2023-09-22)
  
  As I'm new to the AI/ML field, I'm still learning from various online materials. In this particular instance, I've been studying the Reinforcement Learning tutorial by deeplizard, specifically focusing on videos 8 through 10.
  After attempting to replicate the example that demonstrates how to train an agent in the gym's FrozenLake environment, I encountered some unexpected issues. Although I successfully debugged the code and got the algorithm to work, I can't confidently claim that I fully comprehend if this is the correct approach. Therefore, I have specific questions related to this exercise that I intend to post within this single thread. Please feel free to answer any of the questions at your convenience.
  Q1
  What is this gym environment warning all about, and how should it be properly addressed?
  /home/boris/.local/lib/python3.11/site-packages/gym/utils/passive_env_checker.py:233: DeprecationWarning: `np.bool8` is a deprecated alias for `np.bool_`.  (Deprecated NumPy 1.24)
    if not isinstance(terminated, (bool, np.bool8)):
  
  So far, I've discovered two methods to handle this warning: One is to simply  suppress the warning as follows:
  from warnings import filterwarnings
  filterwarnings(action='ignore', category=DeprecationWarning, message='`np.bool8` is a deprecated alias')
  
  Another is to replace the gym environment with the gymnasium environment, which does not produce this warning.
  #import gym
  import gymnasium as gym
  
  This brings me to my second question.
  Q2
  Why are there two environments, gym and gymnasium, that do the same thing?
  Most online examples use gym, but I believe gymnasium is a better choice. Am I mistaken?
  Q3
  What's the distinction between "terminated" and "truncated"? I've made numerous attempts, but I couldn't find a situation where they diverge. They consistently evaluate to either both True or both False.
  In the current 'gym' version, I have to use "FrozenLake-v1" (originally, "FrozenLake-v0" was used in the tutorial), and this modification causes the code to break because the environment now expects 5 values instead of 4. Initially, I fixed this issue as follows:
  new_state, reward, done, UNKNOWN, info = env.step(action)
  
  I discovered later that the correct name for this variable is "truncated." Consequently, I adjusted the code as follows:
  new_state, reward, terminated, truncated, info = env.step(action)
  done = terminated or truncated
  
  Nonetheless, I remain perplexed regarding the usage of "terminated" and "truncated." While experimenting with various algorithm options, such as max_steps_per_episode, I was unable to get them to differ. In all cases, they either both evaluate to True or both to False.
  Q4
  If I want to continue training the agent in stages and test its intermediate performance, how can I do that? I was thinking it's just a matter of saving the q_table, then loading the saved q_table and replacing the current q_table initialization with the new one before resuming training. Or is it not that simple?
  saved_q_table = anydbm.open(filename, 'r')
  
  #q_table = np.zeros((state_space_size, action_space_size))
  q_table = pickle.loads(saved_q_table)
  
  Q5
  Is there a way to get some documentation or help about the environment?
  I'm seeking a resource that I can print and reference while coding to gain a better understanding of the environment's parameters. For instance, in scikit-learn, you can achieve this with the following code:
  from sklearn.neighbors import KNeighborsRegressor
  from pprint import pprint
  pprint(KNeighborsRegressor().get_params())
  
  Resulting in following output:
  {'algorithm': 'auto',
   'leaf_size': 30,
   'metric': 'minkowski',
   'metric_params': None,
   'n_jobs': None,
   'n_neighbors': 5,
   'p': 2,
   'weights': 'uniform'}
  
  It would be great if one could get relevant agent/states parameters from the gym environments in a similar way.
  Q6
  How to use render_mode="human" correctly? Currently, I'm using render_mode="ansi" and rendering the environment as follows:
  env = gym.make("FrozenLake-v1", map_name="8x8", is_slippery=True, render_mode="ansi", desc=None)
  
  print("\033c")
  env.render()
  print(env.render())
  
  However, when I switch to render_mode="human", the environment automatically displays without the need for env.render(). Consequently, the environment renders during training as well, leading to extremely slow training. How can I disable this automatic rendering when using render_mode="human"?
  Q7
  Why doesn't this algorithm work for an 8x8 map size?
  env = gym.make("FrozenLake-v1", map_name="8x8", is_slippery=True, render_mode="ansi", desc=None)
  
  It appears that the agent is being trained, but it fails to learn anything.
  *** Average reward per thousand episodes ***
  
  1000 :  0.0
  2000 :  0.0
  3000 :  0.0
  4000 :  0.0
  5000 :  0.0
  6000 :  0.0
  7000 :  0.0
  8000 :  0.0
  9000 :  0.0
  10000 :  0.0
  
  I can't seem to figure out what needs to be modified. The q_table is the correct size:
  q_table = np.zeros((state_space_size, action_space_size))
  
  I've also increased max_steps_per_episode to 3000. Is it possible that num_episodes = 10000 isn't sufficient?
  By the way, here is the full code I have so far:
  #!/usr/bin/python
  
  # -*- coding: utf-8 -*-
  
  import numpy as np
  #import gym
  import gymnasium as gym
  import random
  import time
  from warnings import filterwarnings
  
  #filterwarnings(action='ignore', category=DeprecationWarning, message='`np.bool8` is a deprecated alias')
  
  env = gym.make("FrozenLake-v1", map_name="4x4", is_slippery=True, render_mode="ansi", desc=None)
  #env = gym.make("FrozenLake-v1", map_name="4x4", is_slippery=True, render_mode="human", desc=None)
  
  action_space_size = env.action_space.n
  state_space_size = env.observation_space.n
  #print(action_space_size)
  #print(state_space_size)
  
  q_table = np.zeros((state_space_size, action_space_size))
  
  num_episodes = 10000
  max_steps_per_episode = 3000
  
  learning_rate = 0.11
  discount_rate = 0.99
  
  exploration_rate = 1
  max_exploration_rate = 0.99
  min_exploration_rate = 0.01
  exploration_decay_rate = 0.002
  
  rewards_all_episodes = []
  
  # Q-Learnig Algorithm
  for episode in range(num_episodes):
      print("*** Episode ",  episode+1, " ***", end="\r")
      state = env.reset()[0]
  
      done = False
      rewards_current_episode = 0
  
      for step in range(max_steps_per_episode):
  
          # Exploration-Exploitation Trade-off
          exploration_rate_threshold = random.uniform(0, 1)
          if exploration_rate_threshold > exploration_rate:
              action = np.argmax(q_table[state, :])
          else:
              action = env.action_space.sample()
          new_state, reward, terminated, truncated, info = env.step(action)
          done = terminated or truncated
  
          # Update Q-Table for Q(s,a)
          q_table[state, action] = q_table[state, action] * (1 - learning_rate) + learning_rate * (reward + discount_rate * np.max(q_table[new_state, :]))
  
          state = new_state
          rewards_current_episode += reward
  
          if done == True:
              break
  
      # Exploration rate decay
      exploration_rate = min_exploration_rate + (max_exploration_rate - min_exploration_rate) * np.exp(-exploration_decay_rate * episode)
  
      rewards_all_episodes.append(rewards_current_episode)
  
  # Calsulate and print the average reward per thousand episodes
  rewards_per_thousand_episodes = np.split(np.array(rewards_all_episodes), num_episodes/1000)
  count = 1000
  print("*** Average reward per thousand episodes ***\n")
  for r in rewards_per_thousand_episodes:
      print(count, ": ", str(sum(r/1000)))
      count += 1000
  
  # Print updated Q-Table
  print("\n\n*** Updated Q-Table ***\n")
  print(q_table)
  input("Press Enter to continue...")
  
  num_episodes = 5
  max_steps_per_episode = 99
  
  for episode in range(num_episodes):
      state = env.reset()[0]
  
      done = False
  
      for step in range(max_steps_per_episode):
          print("\033c")
          print("*** Episode ",  episode+1, " ***\n", step)
          env.render()
          print(env.render())
          time.sleep(0.1)
  
          action = np.argmax(q_table[state, :])
          new_state, reward, done, unknown, info = env.step(action)
  
          # Update Q-Table for Q(s,a)
          q_table[state, action] = q_table[state, action] * (1 - learning_rate) + learning_rate * (reward + discount_rate * np.max(q_table[new_state, :]))
  
          state = new_state
          rewards_current_episode += reward
  
          if done == True:
              print("\033c")
              print("*** Episode ",  episode+1, " ***\n")
              env.render()
              print(env.render())
              if reward == 1:
                  print("*** You reached the goal! ***")
                  time.sleep(3)
              else:
                  print("*** You fell through a hole! ***")
                  time.sleep(3)
              #clear_output(wait=True)
              break
  
          if step == max_steps_per_episode - 1:
              print("*** You failed to complete the task! ***")
              time.sleep(3)
  
  **Tags:** reinforcement-learning, q-learning, gym, frozen-lake
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-09-22. Score: 2
  
  > Q1. What is this gym environment warning all about, and how should it be properly addressed?
  
  The development of `gym` has ended I think. Anyway the error is about an old usage of `numpy`, so probably you have installed a newer version of it.
  
  > Q2. Why are there two environments, gym and gymnasium, that do the same thing?
  
  In principle, they do the same thing but `gym` was developed by OpenAI and `gymnasium` by Farama Foundation, which inherited `gym` basically. You should use `gymnasium` from now on, since is in actual development. This means that initially `gym` and `gymnasium` were almost identical but now their are diverging, even if the underlying concepts are still the same.
  
  > Q3. What's the distinction between "terminated" and "truncated"?
  
  `truncated` indicates that the episode ended due to a time limit, i.e. the current timestep have exceeded the time limit so the episode ends. This also indicated that you have to *bootstrap* the Q value estimate.
  
  `terminated` instead is when the agent have reached a terminal state, therefore the episode naturally ends. You should use both to understand when an episode terminates, e.g. `if truncated or terminated: ...`, but the two flags are useful to determine when or not to bootstrap: you don't do that on termination, for example.
  
  > Q4. If I want to continue training the agent in stages and test its intermediate performance, how can I do that?
  
  You need to save the Q-table (or, in general, the weights/parameters of a function approximator like a neural-net) but also the current value of the `episilon` (if using some schedule, so if changing during the episodes) and similar. Then you instantiate the agent, load the various parameters, and continue training where you left.
  
  > Q5. Is there a way to get some documentation or help about the environment?
  
  Try `help(env)`, the official website (<https://gymnasium.farama.org/index.html>), or read the env's source code.
  
  > Q6. How to use render\_mode="human" correctly?
  
  If you set `render_mode="human"` gymnasium will render at each `step()` and even `reset()`: this is something that `gym` not used to do.
  
  I'm not sure but you may need to wrap the env with this `gymnasium.wrappers.RenderCollection` ([docs](https://gymnasium.farama.org/api/wrappers/misc_wrappers/#gymnasium.wrappers.RenderCollection)) to trigger the rendering only when `render()` is called - I haven't tested that, so I can be wrong.
  
  > **Comments:**
  > **username_0**: Thanks a lot! To summarize,  / - `gymnasium` environments are the way to go  / - `help(env)` prints documentation about environment  / - need to learn about bootstrapping the Q value estimate to use `truncated` flag  / - to resume training need both Q-table and epsilon value  / - check `gymnasium.wrappers` for advanced rendering options  
  > **username_1**: @BorisL. Yeah, that's all you need to do and know in short. Also bootstrapping is something you'll find in Q-learning and TD-learning. I think there is some good answer about it, in AI SE.  
  
  ---
  
  