Body: <p>My OpenAI CartPole-v0 problem's implementation using basic Q-learning does not learn at all.
  I am a beginner and have implemented my first ever Q-learning from scratch after learning from tutorials.</p>
  <p>Can anyone suggest what is going wrong?</p>
  <p>I have seen through testing that the problem may be that most of the states are remain unvisited even after 10,000 runs. Hence, Q-table remains mostly unchanged at the end of all episodes. I have seen other things in the implementation and they all seem fine to me, at least. Any tip where I should start looking at?</p>
  <p>The reward is -200 flat, for all the episodes! which suggests that the improvement is NILL/NADDA/NONE!</p>
  <p>Some relevant images are given at the end.</p>
  <p>The q-learning part of code is given below:</p>
  <pre><code>env.reset()
  while not done:    
      current_state = current_state_to_string(assign_obs_to_bins(obs, bins))
  
      if np.random.uniform() &lt; EPSILON:
          act = env.action_space.sample()
          best_q_value = return_max_from_dict(q[current_state], action = act)
      else:
          act, best_q_value = return_max_from_dict(q[current_state])
  
      obs, reward, done, _  = env.step(act)
      q[current_state][act] += LEARNING_RATE * (reward + DISCOUNT_FACTOR * best_q_value - q[current_state][act])
      cnt+=1
      total_reward += reward
  </code></pre>
  <p><a href="https://i.sstatic.net/pbcpG.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/pbcpG.jpg" alt="States' Histogram" /></a></p>
  <p><a href="https://i.sstatic.net/O5RGs.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/O5RGs.jpg" alt="Reward after all episodes" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-27T22:24:04.117Z
Comments.Text: I would like to add one more thing that I tried epsilon-greed approach, greedy approach, and random approach. All gave me the same result in terms of reward and in all cases, most of the states remained untouched.
Comments.UserDisplayName: SJa
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-27T23:49:33.927Z
Comments.Text: Why are you using Q learning rather than DQN?
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-28T05:48:42.620Z
Comments.Text: because I am a beginner and do not want to ride a sports bike before properly learning to balance on the bicycle :)
Comments.UserDisplayName: SJa
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-28T06:45:17.200Z
Comments.Text: try to test your implementation on a simple grid world and see if it works. After that you need to properly discretize state space. Try using simple case first, use only 2 states, one state when pole is to the left of the 90 degrees, and second when its right to 90 degrees. Then keep refining your discretization if your performance is not good enough.
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-28T08:49:09.717Z
Comments.Text: CartPole will not be solvable with tabular DQN without some tuning of the state space as @brale mentioned. Just do some of the exercises from the Sutton and Barto book if you want to practice Tabular Q-Learning.
Comments.UserDisplayName: David
Tags: <q-learning><open-ai><gym>
Title: OpenAI gym's CartPole problem system does not learn
UserDisplayName: SJa
ThreadText: # OpenAI gym's CartPole problem system does not learn
  
  (Asked by: username_0 on 2020-07-27)
  
  My OpenAI CartPole-v0 problem's implementation using basic Q-learning does not learn at all.
  I am a beginner and have implemented my first ever Q-learning from scratch after learning from tutorials.
  
  Can anyone suggest what is going wrong?
  
  I have seen through testing that the problem may be that most of the states are remain unvisited even after 10,000 runs. Hence, Q-table remains mostly unchanged at the end of all episodes. I have seen other things in the implementation and they all seem fine to me, at least. Any tip where I should start looking at?
  
  The reward is -200 flat, for all the episodes! which suggests that the improvement is NILL/NADDA/NONE!
  
  Some relevant images are given at the end.
  
  The q-learning part of code is given below:
  
  ```
  env.reset()
  while not done:    
      current_state = current_state_to_string(assign_obs_to_bins(obs, bins))
  
      if np.random.uniform() < EPSILON:
          act = env.action_space.sample()
          best_q_value = return_max_from_dict(q[current_state], action = act)
      else:
          act, best_q_value = return_max_from_dict(q[current_state])
  
      obs, reward, done, _  = env.step(act)
      q[current_state][act] += LEARNING_RATE * (reward + DISCOUNT_FACTOR * best_q_value - q[current_state][act])
      cnt+=1
      total_reward += reward
  
  ```
  
  [![States' Histogram](https://i.sstatic.net/pbcpG.jpg)](https://i.sstatic.net/pbcpG.jpg)
  
  [![Reward after all episodes](https://i.sstatic.net/O5RGs.jpg)](https://i.sstatic.net/O5RGs.jpg)
  
  **Tags:** q-learning, open-ai, gym
  
  > **Comments:**
  > **username_0**: I would like to add one more thing that I tried epsilon-greed approach, greedy approach, and random approach. All gave me the same result in terms of reward and in all cases, most of the states remained untouched.  
  > **username_1**: Why are you using Q learning rather than DQN?  
  > **username_0**: because I am a beginner and do not want to ride a sports bike before properly learning to balance on the bicycle :)  
  > **username_2**: try to test your implementation on a simple grid world and see if it works. After that you need to properly discretize state space. Try using simple case first, use only 2 states, one state when pole is to the left of the 90 degrees, and second when its right to 90 degrees. Then keep refining your discretization if your performance is not good enough.  
  > **username_1**: CartPole will not be solvable with tabular DQN without some tuning of the state space as @username_2 mentioned. Just do some of the exercises from the Sutton and Barto book if you want to practice Tabular Q-Learning.  
  
  ---
  
  ## 0 answers
  
  