Answers.Body: <p>The problem here is likely related to the state approximations you are using.</p>
  
  <p>Unfortunately, OpenAI's gym does not always give reasonable bounds when using <code>env.observation_space</code>, and that seems to be the case for CartPole:</p>
  
  <pre><code>&gt;&gt;&gt; env = gym.make('CartPole-v0')
  &gt;&gt;&gt; env.observation_space.high
  array([4.8000002e+00, 3.4028235e+38, 4.1887903e-01, 3.4028235e+38],
        dtype=float32)
  &gt;&gt;&gt; env.observation_space.low
  array([-4.8000002e+00, -3.4028235e+38, -4.1887903e-01, -3.4028235e+38],
        dtype=float32)
  </code></pre>
  
  <p>Processing this, similarly to your code:</p>
  
  <pre><code>&gt;&gt;&gt; discrete_os_win_size = (env.observation_space.high - env.observation_space.low)/ DISCRETE_OS_SIZE
  __main__:1: RuntimeWarning: overflow encountered in subtract
  &gt;&gt;&gt; discrete_os_win_size
  array([0.48000002,        inf, 0.0418879 ,        inf])
  
  &gt;&gt;&gt; discrete_state = (state - env.observation_space.low)/discrete_os_win_size
  &gt;&gt;&gt; discrete_state
  array([11.27318768,  0.        , 19.50682776,  0.        ])
  </code></pre>
  
  <p>That means that all the velocities will get squashed down to <span class="math-container">$0$</span> in your approximation. Your agent cannot tell the difference between a nearly static balancing position (generally the goal) and transitioning through it really fast - it will think that both are equally good. It is also not able to tell difference between moving towards balance point, or moving away from it.</p>
  
  <p>I suggest you check for what reasonable bounds are on the space (a quick look suggests +/- 2.0 might be a reasonable starting point) and use that instead.</p>
  
  <p>The approximation approach of discrete grid is also very crude, although it does allow you do use tabular approaches. If you want to stick with a linear system (and avoid trying neural networks and DQN) then the next step up would be <a href="https://towardsdatascience.com/reinforcement-learning-tile-coding-implementation-7974b600762b" rel="nofollow noreferrer">some form of tile coding</a>, which uses multiple offset grids to obtain smoother interpolation between states.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-14T15:21:41.910Z
Answers.Text: This is fantastic thank you! It's not that I want to stick with a linear system I just want to tackle similar problems with various approaches to help me understand which approaches work best where, and more importantly why. From all of this I have learn that tabular approaches are not useful for continuous spaces.
Answers.UserDisplayName: BlueTurtle
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-14T14:49:13.527Z
Answers.LastActivityDate: 2020-06-14T15:17:37.177Z
Answers.LastEditDate: 2020-06-14T15:17:37.177Z
Answers.UserDisplayName: Neil Slater
Body: <p>Using the tutorial from: <a href="https://pythonprogramming.net/q-learning-algorithm-reinforcement-learning-python-tutorial/?completed=/q-learning-reinforcement-learning-python-tutorial/" rel="nofollow noreferrer">SentDex - Python Programming</a> I added Q Learning to my script that was previously just picking random actions. His script uses the MountainCar Environment so I had to amend it to the CartPole env I am using. Initially, the rewards seem sporadic but, after a while, they just drop off and oscillate between 0-10. Does anyone know why this is?</p>
  
  <pre><code>Learning_rate = 0.1
  Discount_rate = 0.95
  episodes = 200
  
  # Exploration settings
  epsilon = 1  # not a constant, qoing to be decayed
  START_EPSILON_DECAYING = 1
  END_EPSILON_DECAYING = episodes//2
  epsilon_decay_value = epsilon/(END_EPSILON_DECAYING - START_EPSILON_DECAYING)
  
  env = gym.make("CartPole-v0") #Create the environment. The name of the environments can be found @ https://gym.openai.com/envs/#classic_control
  #Each environment has a number of possible actions. In this case there are two discrete actions, left or right
  
  #Each environment has some integer characteristics of the state.
  #In this case we have 4:
  
  #env = gym.wrappers.Monitor(env, './', force=True)
  
  DISCRETE_OS_SIZE = [20, 20, 20, 20]
  
  discrete_os_win_size = (env.observation_space.high - env.observation_space.low)/ DISCRETE_OS_SIZE 
  
  def get_discrete_state(state):
      discrete_state = (state - env.observation_space.low)/discrete_os_win_size
      return tuple(discrete_state.astype(np.int))
  
  q_table = np.random.uniform(low = -2, high = 0, size = (20, 20, 20, 20, env.action_space.n))
  
  plt.figure() #Instantiate the plotting environment
  rewards_list = [] #Create an empty list to add the rewards to which we will then plot
  for i in range(episodes):
      discrete_state = get_discrete_state(env.reset())
      done = False
      rewards = 0
      frames = []
  
      while not done:
          #frames.append(env.render(mode = "rgb_array"))
  
          if np.random.random() &gt; epsilon:
              # Get action from Q table
              action = np.argmax(q_table[discrete_state])
  
          else:
              # Get random action
              action = np.random.randint(0, env.action_space.n)
  
          new_state, reward, done, info = env.step(action)
  
          new_discrete_state = get_discrete_state(new_state)
  
          # If simulation did not end yet after last step - update Q table
          if not done:
  
              # Maximum possible Q value in next step (for new state)
              max_future_q = np.max(q_table[new_discrete_state])
  
              # Current Q value (for current state and performed action)
              current_q = q_table[discrete_state, action]
  
              # And here's our equation for a new Q value for current state and action
              new_q = (1 - Learning_rate) * current_q + Learning_rate * (reward + Discount_rate * max_future_q)
  
              # Update Q table with new Q value
              q_table[discrete_state, action] = new_q
  
          else:
              q_table[discrete_state + (action,)] = 0
  
          discrete_state = new_discrete_state
  
          rewards += reward
          rewards_list.append(rewards)
      #print("Episode:", i, "Rewards:", rewards)
      #print("Observations:", obs)
  
      # Decaying is being done every episode if episode number is within decaying range
      if END_EPSILON_DECAYING &gt;= i &gt;= START_EPSILON_DECAYING:
          epsilon -= epsilon_decay_value
  
  plt.plot(rewards_list)
  plt.show()
  env.close()
  </code></pre>
  
  <p><a href="https://i.sstatic.net/0Abvo.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0Abvo.png" alt="CartPoleRewards"></a></p>
  
  <p>It becomes even more pronounced when I increase episodes to 20,000 so I don't think it's related to not giving the model enough training time.</p>
  
  <p><a href="https://i.sstatic.net/LaVby.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/LaVby.png" alt="20000 episodes"></a></p>
  
  <p>If I set <code>START_EPSILON_DECAYING</code> to say 200 then it only drops to &lt; 10 rewards after episode 200 which made me think it was the epsilon that was causing the problem. However, if I remove the epsilon/exploratory then the rewards at every episode are worse as it gets stuck in picking the argmax value for each state.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-14T14:15:36.813Z
Comments.Text: Note that our site focuses more on answering theoretical questions. Please, read our on-topic page: https://ai.stackexchange.com/help/on-topic. Your question could just be a bug in the code, so that would be considered off-topic. Nevertheless, given that this is related to RL, which is a very important topic in AI, then I will not close this post.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-14T14:20:43.947Z
Comments.Text: The guys on Stack Overflow don't know the details of our scope. That's why I suggest you read our on-topic page. Your post is on-topic here. But if it turns out that your problem is just a programming bug, then that could be considered off-topic (but don't worry about that!). If you read our on-topic page, hopefully, this will become clearer. Please, let me know if our on-topic page is clear enough!
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-14T14:21:38.147Z
Comments.Text: @nbro: Looks like classic catastrophic forgetting, and I think we already have a Q&A for that that this is a duplicate for - I'm searching now
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-14T14:23:43.280Z
Comments.Text: @nbro Thank you. I have looked at the on-topic page and see that it suggests: "General programming questions are off-topic. For example, if you have a question like "Why am I getting this exception?" So I agree this is probably more appropriate on SO. Should I delete the post here or post there also? I don't want to create too much clutter.
Comments.UserDisplayName: BlueTurtle
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-14T14:24:19.850Z
Comments.Text: @NeilSlater I hadn't heard of that term, probably why my search didn't return much of any use. I will look into it now using that keyword. However, any specific links would be appreciated.
Comments.UserDisplayName: BlueTurtle
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-14T14:26:22.560Z
Comments.Text: @DanielTJones Actually the "duplicate" is on Data Science (I answered there): https://datascience.stackexchange.com/questions/56053/why-could-my-ddqn-get-significantly-worse-after-beating-the-game-repeatedly/56055#56055
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-14T14:31:15.737Z
Comments.Text: @DanielTJones You're not getting any exception or error, so that's not apparently a programming bug.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-14T14:32:00.290Z
Comments.Text: @NeilSlater Many thanks. I am new(ish) to all of this so I may be wrong but given that I am using just a basic Q Table lookup rather than a DDQN then is the catastrophic forgetting applicable? I am now wondering if it's a numerical error somewhere that is getting propegated through over episodes and distorting my Q values. It seems strange that it starts off okay but then diminishes. Compared to that post which increases and then decreases.
Comments.UserDisplayName: BlueTurtle
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-14T14:33:38.673Z
Comments.Text: @NeilSlater Sorry the quote was just a snippet of the more general point the bullet is making that I clipped. The bullet is making the general point of anything coding related may be off-topic. My poor clipping skills.
Comments.UserDisplayName: BlueTurtle
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-14T14:36:59.633Z
Comments.Text: @DanielTJones: The environment has contnuous state, so you have approximation (looks like quite crude approximation), and that can still have problems with forgetting. However, maybe you are right, it isn't an issue here
Comments.UserDisplayName: Neil Slater
Tags: <reinforcement-learning><q-learning>
Title: Why do my rewards fall using tabular Q-learning as I perform more episodes?
UserDisplayName: BlueTurtle
ThreadText: # Why do my rewards fall using tabular Q-learning as I perform more episodes?
  
  (Asked by: username_0 on 2020-06-14)
  
  Using the tutorial from: [SentDex - Python Programming](https://pythonprogramming.net/q-learning-algorithm-reinforcement-learning-python-tutorial/?completed=/q-learning-reinforcement-learning-python-tutorial/) I added Q Learning to my script that was previously just picking random actions. His script uses the MountainCar Environment so I had to amend it to the CartPole env I am using. Initially, the rewards seem sporadic but, after a while, they just drop off and oscillate between 0-10. Does anyone know why this is?
  
  ```
  Learning_rate = 0.1
  Discount_rate = 0.95
  episodes = 200
  
  # Exploration settings
  epsilon = 1  # not a constant, qoing to be decayed
  START_EPSILON_DECAYING = 1
  END_EPSILON_DECAYING = episodes//2
  epsilon_decay_value = epsilon/(END_EPSILON_DECAYING - START_EPSILON_DECAYING)
  
  env = gym.make("CartPole-v0") #Create the environment. The name of the environments can be found @ https://gym.openai.com/envs/#classic_control
  #Each environment has a number of possible actions. In this case there are two discrete actions, left or right
  
  #Each environment has some integer characteristics of the state.
  #In this case we have 4:
  
  #env = gym.wrappers.Monitor(env, './', force=True)
  
  DISCRETE_OS_SIZE = [20, 20, 20, 20]
  
  discrete_os_win_size = (env.observation_space.high - env.observation_space.low)/ DISCRETE_OS_SIZE 
  
  def get_discrete_state(state):
      discrete_state = (state - env.observation_space.low)/discrete_os_win_size
      return tuple(discrete_state.astype(np.int))
  
  q_table = np.random.uniform(low = -2, high = 0, size = (20, 20, 20, 20, env.action_space.n))
  
  plt.figure() #Instantiate the plotting environment
  rewards_list = [] #Create an empty list to add the rewards to which we will then plot
  for i in range(episodes):
      discrete_state = get_discrete_state(env.reset())
      done = False
      rewards = 0
      frames = []
  
      while not done:
          #frames.append(env.render(mode = "rgb_array"))
  
          if np.random.random() > epsilon:
              # Get action from Q table
              action = np.argmax(q_table[discrete_state])
  
          else:
              # Get random action
              action = np.random.randint(0, env.action_space.n)
  
          new_state, reward, done, info = env.step(action)
  
          new_discrete_state = get_discrete_state(new_state)
  
          # If simulation did not end yet after last step - update Q table
          if not done:
  
              # Maximum possible Q value in next step (for new state)
              max_future_q = np.max(q_table[new_discrete_state])
  
              # Current Q value (for current state and performed action)
              current_q = q_table[discrete_state, action]
  
              # And here's our equation for a new Q value for current state and action
              new_q = (1 - Learning_rate) * current_q + Learning_rate * (reward + Discount_rate * max_future_q)
  
              # Update Q table with new Q value
              q_table[discrete_state, action] = new_q
  
          else:
              q_table[discrete_state + (action,)] = 0
  
          discrete_state = new_discrete_state
  
          rewards += reward
          rewards_list.append(rewards)
      #print("Episode:", i, "Rewards:", rewards)
      #print("Observations:", obs)
  
      # Decaying is being done every episode if episode number is within decaying range
      if END_EPSILON_DECAYING >= i >= START_EPSILON_DECAYING:
          epsilon -= epsilon_decay_value
  
  plt.plot(rewards_list)
  plt.show()
  env.close()
  
  ```
  
  [![CartPoleRewards](https://i.sstatic.net/0Abvo.png)](https://i.sstatic.net/0Abvo.png)
  
  It becomes even more pronounced when I increase episodes to 20,000 so I don't think it's related to not giving the model enough training time.
  
  [![20000 episodes](https://i.sstatic.net/LaVby.png)](https://i.sstatic.net/LaVby.png)
  
  If I set `START_EPSILON_DECAYING` to say 200 then it only drops to < 10 rewards after episode 200 which made me think it was the epsilon that was causing the problem. However, if I remove the epsilon/exploratory then the rewards at every episode are worse as it gets stuck in picking the argmax value for each state.
  
  **Tags:** reinforcement-learning, q-learning
  
  > **Comments:**
  > **username_1**: Note that our site focuses more on answering theoretical questions. Please, read our on-topic page: https://ai.stackexchange.com/help/on-topic. Your question could just be a bug in the code, so that would be considered off-topic. Nevertheless, given that this is related to RL, which is a very important topic in AI, then I will not close this post.  
  > **username_1**: The guys on Stack Overflow don't know the details of our scope. That's why I suggest you read our on-topic page. Your post is on-topic here. But if it turns out that your problem is just a programming bug, then that could be considered off-topic (but don't worry about that!). If you read our on-topic page, hopefully, this will become clearer. Please, let me know if our on-topic page is clear enough!  
  > **username_2**: @username_1: Looks like classic catastrophic forgetting, and I think we already have a Q&A for that that this is a duplicate for - I'm searching now  
  > **username_0**: @username_1 Thank you. I have looked at the on-topic page and see that it suggests: "General programming questions are off-topic. For example, if you have a question like "Why am I getting this exception?" So I agree this is probably more appropriate on SO. Should I delete the post here or post there also? I don't want to create too much clutter.  
  > **username_0**: @username_2 I hadn't heard of that term, probably why my search didn't return much of any use. I will look into it now using that keyword. However, any specific links would be appreciated.  
  > **username_2**: @DanielTJones Actually the "duplicate" is on Data Science (I answered there): https://datascience.stackexchange.com/questions/56053/why-could-my-ddqn-get-significantly-worse-after-beating-the-game-repeatedly/56055#56055  
  > **username_1**: @DanielTJones You're not getting any exception or error, so that's not apparently a programming bug.  
  > **username_0**: @username_2 Many thanks. I am new(ish) to all of this so I may be wrong but given that I am using just a basic Q Table lookup rather than a DDQN then is the catastrophic forgetting applicable? I am now wondering if it's a numerical error somewhere that is getting propegated through over episodes and distorting my Q values. It seems strange that it starts off okay but then diminishes. Compared to that post which increases and then decreases.  
  > **username_0**: @username_2 Sorry the quote was just a snippet of the more general point the bullet is making that I clipped. The bullet is making the general point of anything coding related may be off-topic. My poor clipping skills.  
  > **username_2**: @DanielTJones: The environment has contnuous state, so you have approximation (looks like quite crude approximation), and that can still have problems with forgetting. However, maybe you are right, it isn't an issue here  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-06-14. Score: 2 | Accepted answer
  
  The problem here is likely related to the state approximations you are using.
  
  Unfortunately, OpenAI's gym does not always give reasonable bounds when using `env.observation_space`, and that seems to be the case for CartPole:
  
  ```
  >>> env = gym.make('CartPole-v0')
  >>> env.observation_space.high
  array([4.8000002e+00, 3.4028235e+38, 4.1887903e-01, 3.4028235e+38],
        dtype=float32)
  >>> env.observation_space.low
  array([-4.8000002e+00, -3.4028235e+38, -4.1887903e-01, -3.4028235e+38],
        dtype=float32)
  
  ```
  
  Processing this, similarly to your code:
  
  ```
  >>> discrete_os_win_size = (env.observation_space.high - env.observation_space.low)/ DISCRETE_OS_SIZE
  __main__:1: RuntimeWarning: overflow encountered in subtract
  >>> discrete_os_win_size
  array([0.48000002,        inf, 0.0418879 ,        inf])
  
  >>> discrete_state = (state - env.observation_space.low)/discrete_os_win_size
  >>> discrete_state
  array([11.27318768,  0.        , 19.50682776,  0.        ])
  
  ```
  
  That means that all the velocities will get squashed down to $0$ in your approximation. Your agent cannot tell the difference between a nearly static balancing position (generally the goal) and transitioning through it really fast - it will think that both are equally good. It is also not able to tell difference between moving towards balance point, or moving away from it.
  
  I suggest you check for what reasonable bounds are on the space (a quick look suggests +/- 2.0 might be a reasonable starting point) and use that instead.
  
  The approximation approach of discrete grid is also very crude, although it does allow you do use tabular approaches. If you want to stick with a linear system (and avoid trying neural networks and DQN) then the next step up would be [some form of tile coding](https://towardsdatascience.com/reinforcement-learning-tile-coding-implementation-7974b600762b), which uses multiple offset grids to obtain smoother interpolation between states.
  
  > **Comments:**
  > **username_0**: This is fantastic thank you! It's not that I want to stick with a linear system I just want to tackle similar problems with various approaches to help me understand which approaches work best where, and more importantly why. From all of this I have learn that tabular approaches are not useful for continuous spaces.  
  
  ---
  
  