Answers.Body: <p>This isn't possible with a basic tabular approach, or with any agent that has no internal memory. The velocities are a necessary part of the state. Without them, the state is only partially observable, and the difference is really important. For example, imagine the positions are neutral - the pole and cart are in the middle - there's a huge difference between the pole being still or moving fast in either direction. It would change which action was best. This is not universally true for all environments, but a good rule of thumb is if your actions are to apply forces (and therefore impulse or acceleration), then you will need to observe the first-order effect of the actions, i.e. the velocities.</p>
  <p>The Q table that is learned by the agent as you currently have it set up will contain the mean action values that cover the distributions of the pole moving in either direction, for the best policy that the agent can come up with when it is not sure which direction the pole is moving in. This may still be optimal in some sense for the partially-observed state, but it will fall short of &quot;solving&quot; the environment.</p>
  <p>How to address it:</p>
  <ul>
  <li><p>Simplest fix - add the velocity back into the state.</p>
  </li>
  <li><p>Interesting, but hard, fix - use a function approximator with in-built memory for the Q network, such as an LSTM recurrent neural network. Having a memory will allow the agent to figure out velocity and the rules for it due to history of how the pendulum has moved.</p>
  </li>
  </ul>
  <p>The memory version may also be less effective than adding to the state, since the agent needs to observe at least one extra time step before it can approximate the velocity and determine the best action. However, the reason I say it is &quot;interesting&quot; is because it is a setup where the agent really does learn how to solve the harder challenge you have set it.</p>
  <p>In addition, you may find policy gradient methods do slightly better than tabular Q learning, at least in terms of average return. This is because a policy gradient method can learn to &quot;hedge its bets&quot; and behave randomly when it lacks enough state information to make a clear decision.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-03T20:20:25.400Z
Answers.LastActivityDate: 2023-12-03T21:18:55.287Z
Answers.LastEditDate: 2023-12-03T21:18:55.287Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>Thanks man, I already solved the problem with Tabular Q-Learning without Neural Networks. I had to just add the cart and pole velocities to the state and the agent learn to balance the pendulum.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-17T08:21:39.913Z
Answers.Text: No problem. The usual way to show an answer helped you is to select the "tick" mark on the left t of the best answer. That also will give you a little rep.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-16T22:54:24.383Z
Answers.LastActivityDate: 2024-01-16T22:54:24.383Z
Answers.UserDisplayName: Vitor Martins
Body: <p>I'm trying to solve the cartpole-v1 gym environment with only the linear and angular position, but the mean reward of the last 100 episodes isn't greater than 20 rewards. The longest train i made was a train with 90 000 episodes and the agent didn't get more than 20 reward.</p>
  <p>The algorithm that i'm using is the tabular method Q-Learning and Epsilon-greedy for action selection.</p>
  <p>This is the code, i implement:</p>
  <pre><code>import gymnasium as gym
  import numpy as np
  import math
  
  # Axis dimensions
  max_x = 2.4
  min_x = -2.4
  max_theta = 12
  min_theta = -12
  x_bins = 24
  theta_bins = 24
  
  x_axis_space = np.linspace(min_x, max_x, x_bins)
  theta_axis_space = np.linspace(min_theta, max_theta, theta_bins)
  
  # Env setup - First state
  env = gym.make('CartPole-v1')
  obs, info = env.reset() 
  
  last_state = state = (np.digitize(obs[0], x_axis_space), np.digitize(obs[2]*180/math.pi, theta_axis_space))
  
  # import ipdb; ipdb.set_trace()
  # print(state)
  
  # Hyperparameters
  GAMMA = 0.99        # Discount factor (0.95 Looking harder for Long-term reward)
  ALPHA = 0.1         # Learning rate
  EPSILON = 1         # 100% of exploration
  N_EPISODES = 4000
  MAX_REWARD = 500
  total_reward = 0
  episode_reward = []
  EPSILON = 1.0 
  DECAY_RATIO = 1-0.00001
  
  
  class CartPoleQAgent():
      def __init__(self, n_bins_x, n_bins_theta, n_actions):       # x  theta L//R
          self.n_bins_x = n_bins_x
          self.n_bins_theta = n_bins_theta
          self.n_actions = n_actions
          self.q_table = np.zeros((n_bins_x+1, n_bins_theta+1, n_actions))
  
  
  def exp_dec_epsilon_greedy(q_table, state, finish_training):
      global EPSILON
      
      if np.random.random() &gt; EPSILON or finish_training == 1:    # Exploit
          # Select the greedy action max Q
          max_q = q_table[state[0]][state[1]].max()
          for i in range(2):
              if max_q == q_table[state[0]][state[1]][i]:
                  return i
      else:                                                       # Explore
          # Select a random action
          return env.action_space.sample()
  
  
  def update_q_value(q_table, last_state, action, reward, state):
      global GAMMA, ALPHA
      # import ipdb; ipdb.set_trace()
      action = int(action)
      last_x = last_state[0]  
      last_theta = last_state[1]
      x = state[0] 
      theta = state[1]
  
      return (q_table[last_x][last_theta][action] + ALPHA*(reward + GAMMA*q_table[x][theta][action] - q_table[last_x][last_theta][action]))
  
  
  if __name__ == &quot;__main__&quot;:
      
      agent = CartPoleQAgent(24, 24, 2)
      finish_training = 0
      i_episode = 0
      mean_reward = 0
      
      while mean_reward &lt; 500:
          i_episode_reward = 0  
          
          while True:     # End of an episode
              action = exp_dec_epsilon_greedy(agent.q_table, state, finish_training)
              
              result = env.step(action)  
              obs, reward, done, info = result[:4]
              i_episode_reward = i_episode_reward + reward
  
              if done:    # If the episode has ended
                  env.reset() # Always the cartpole end conditions are met, to reboot the env
                  break
              
              state = (np.digitize(obs[0], x_axis_space), np.digitize(obs[2]*180/math.pi, theta_axis_space))
              agent.q_table[last_state[0]][last_state[1]][action] = update_q_value(agent.q_table, last_state, action, reward, state)
              last_state = state
          
          episode_reward.append(i_episode_reward)
          EPSILON = EPSILON * DECAY_RATIO 
  
          mean_reward = np.mean(episode_reward[len(episode_reward)-100:])    
          print(&quot;Episode: &quot; + str(i_episode) + &quot; Episode Reward: &quot; + str(i_episode_reward) + &quot; eps: &quot; + str(EPSILON) + &quot; Mean Reward: &quot; + str(mean_reward))
          i_episode = i_episode + 1 
      
  env.close()
  </code></pre>
  
Tags: <reinforcement-learning><q-learning>
Title: Can we solve the environment with only the linear and angular position through Q-Learning?
UserDisplayName: Vitor Martins
ThreadText: # Can we solve the environment with only the linear and angular position through Q-Learning?
  
  (Asked by: username_0 on 2023-12-03)
  
  I'm trying to solve the cartpole-v1 gym environment with only the linear and angular position, but the mean reward of the last 100 episodes isn't greater than 20 rewards. The longest train i made was a train with 90 000 episodes and the agent didn't get more than 20 reward.
  The algorithm that i'm using is the tabular method Q-Learning and Epsilon-greedy for action selection.
  This is the code, i implement:
  import gymnasium as gym
  import numpy as np
  import math
  
  # Axis dimensions
  max_x = 2.4
  min_x = -2.4
  max_theta = 12
  min_theta = -12
  x_bins = 24
  theta_bins = 24
  
  x_axis_space = np.linspace(min_x, max_x, x_bins)
  theta_axis_space = np.linspace(min_theta, max_theta, theta_bins)
  
  # Env setup - First state
  env = gym.make('CartPole-v1')
  obs, info = env.reset() 
  
  last_state = state = (np.digitize(obs[0], x_axis_space), np.digitize(obs[2]*180/math.pi, theta_axis_space))
  
  # import ipdb; ipdb.set_trace()
  # print(state)
  
  # Hyperparameters
  GAMMA = 0.99        # Discount factor (0.95 Looking harder for Long-term reward)
  ALPHA = 0.1         # Learning rate
  EPSILON = 1         # 100% of exploration
  N_EPISODES = 4000
  MAX_REWARD = 500
  total_reward = 0
  episode_reward = []
  EPSILON = 1.0 
  DECAY_RATIO = 1-0.00001
  
  
  class CartPoleQAgent():
      def __init__(self, n_bins_x, n_bins_theta, n_actions):       # x  theta L//R
          self.n_bins_x = n_bins_x
          self.n_bins_theta = n_bins_theta
          self.n_actions = n_actions
          self.q_table = np.zeros((n_bins_x+1, n_bins_theta+1, n_actions))
  
  
  def exp_dec_epsilon_greedy(q_table, state, finish_training):
      global EPSILON
      
      if np.random.random() > EPSILON or finish_training == 1:    # Exploit
          # Select the greedy action max Q
          max_q = q_table[state[0]][state[1]].max()
          for i in range(2):
              if max_q == q_table[state[0]][state[1]][i]:
                  return i
      else:                                                       # Explore
          # Select a random action
          return env.action_space.sample()
  
  
  def update_q_value(q_table, last_state, action, reward, state):
      global GAMMA, ALPHA
      # import ipdb; ipdb.set_trace()
      action = int(action)
      last_x = last_state[0]  
      last_theta = last_state[1]
      x = state[0] 
      theta = state[1]
  
      return (q_table[last_x][last_theta][action] + ALPHA*(reward + GAMMA*q_table[x][theta][action] - q_table[last_x][last_theta][action]))
  
  
  if __name__ == "__main__":
      
      agent = CartPoleQAgent(24, 24, 2)
      finish_training = 0
      i_episode = 0
      mean_reward = 0
      
      while mean_reward < 500:
          i_episode_reward = 0  
          
          while True:     # End of an episode
              action = exp_dec_epsilon_greedy(agent.q_table, state, finish_training)
              
              result = env.step(action)  
              obs, reward, done, info = result[:4]
              i_episode_reward = i_episode_reward + reward
  
              if done:    # If the episode has ended
                  env.reset() # Always the cartpole end conditions are met, to reboot the env
                  break
              
              state = (np.digitize(obs[0], x_axis_space), np.digitize(obs[2]*180/math.pi, theta_axis_space))
              agent.q_table[last_state[0]][last_state[1]][action] = update_q_value(agent.q_table, last_state, action, reward, state)
              last_state = state
          
          episode_reward.append(i_episode_reward)
          EPSILON = EPSILON * DECAY_RATIO 
  
          mean_reward = np.mean(episode_reward[len(episode_reward)-100:])    
          print("Episode: " + str(i_episode) + " Episode Reward: " + str(i_episode_reward) + " eps: " + str(EPSILON) + " Mean Reward: " + str(mean_reward))
          i_episode = i_episode + 1 
      
  env.close()
  
  **Tags:** reinforcement-learning, q-learning
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-12-03. Score: 3
  
  This isn't possible with a basic tabular approach, or with any agent that has no internal memory. The velocities are a necessary part of the state. Without them, the state is only partially observable, and the difference is really important. For example, imagine the positions are neutral - the pole and cart are in the middle - there's a huge difference between the pole being still or moving fast in either direction. It would change which action was best. This is not universally true for all environments, but a good rule of thumb is if your actions are to apply forces (and therefore impulse or acceleration), then you will need to observe the first-order effect of the actions, i.e. the velocities.
  
  The Q table that is learned by the agent as you currently have it set up will contain the mean action values that cover the distributions of the pole moving in either direction, for the best policy that the agent can come up with when it is not sure which direction the pole is moving in. This may still be optimal in some sense for the partially-observed state, but it will fall short of "solving" the environment.
  
  How to address it:
  
  * Simplest fix - add the velocity back into the state.
  * Interesting, but hard, fix - use a function approximator with in-built memory for the Q network, such as an LSTM recurrent neural network. Having a memory will allow the agent to figure out velocity and the rules for it due to history of how the pendulum has moved.
  
  The memory version may also be less effective than adding to the state, since the agent needs to observe at least one extra time step before it can approximate the velocity and determine the best action. However, the reason I say it is "interesting" is because it is a setup where the agent really does learn how to solve the harder challenge you have set it.
  
  In addition, you may find policy gradient methods do slightly better than tabular Q learning, at least in terms of average return. This is because a policy gradient method can learn to "hedge its bets" and behave randomly when it lacks enough state information to make a clear decision.
  
  
  ---
  
  ### Answer by username_0 on 2024-01-16. Score: 0
  
  Thanks man, I already solved the problem with Tabular Q-Learning without Neural Networks. I had to just add the cart and pole velocities to the state and the agent learn to balance the pendulum.
  
  > **Comments:**
  > **username_1**: No problem. The usual way to show an answer helped you is to select the "tick" mark on the left t of the best answer. That also will give you a little rep.  
  
  ---
  
  