Answers.Body: <p>I think the problem is with openAI gym CartPole-v0 environment reward structure.
  The reward is always +1 for each time step. So if pole falls reward is +1 itself.
  So we need to check and redefine the reward for this case.
  So in the train function try this:</p>
  <pre><code>if not done:
      new_q = reward + DISCOUNT * np.max(future_qs_list)
  else:
      # if done assign some negative reward
      new_q = -20
  </code></pre>
  <p>(Or change the reward during replay buffer update)</p>
  <p>Check the lines 81 and 82 in Qlearning.py code in <a href="https://github.com/girishdhegde/reinforcement-learning/" rel="nofollow noreferrer">this</a> repo for further clarification.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-15T22:53:21.967Z
Answers.Text: You really don't need to edit the reward structure. The reward structure is fine. Such a high negative reward can even have negative effects.
Answers.UserDisplayName: hal9000
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-15T23:12:49.777Z
Answers.Text: @hal I’m not sure that that’s true (that you don’t need to change it). If the environment gives +1 on termination then that can be problematic for the agent, it is better to change it to 0 or -1.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-15T23:15:48.460Z
Answers.Text: yeah, I have to agree on that, 0 or -1 seems reasonable to me too but the reward structure is fine enough to be able to solve without doing that.
Answers.UserDisplayName: hal9000
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-16T04:36:08.750Z
Answers.Text: @hal I experimented with different reward stuctures for cartpole using classical Q-learning(Not DQN). I found that reward between -10 to -20 for terminal state working better. My experimentation code and graph can be found [here](https://github.com/girishdhegde/reinforcement-learning/tree/master). But I found [this](https://gym.openai.com/evaluations/eval_aHf1Kmc4QIKm5oPcJJToBA/) code having reward of -200 for terminal state in official gym website.
Answers.UserDisplayName: Girish Hegde
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-15T15:57:09.080Z
Answers.LastActivityDate: 2020-08-16T04:37:40.397Z
Answers.LastEditDate: 2020-08-16T04:37:40.397Z
Answers.UserDisplayName: Girish Hegde
Answers.Body: <p>There is a really small mistake in here that causes the problem: <br/></p>
  <pre><code>
  for index, (current_state, action, reward, next_state, done) in enumerate(minibatch):
              if not done:
                  new_q = reward + DISCOUNT * np.max(future_qs_list) #HERE 
              else:
                  new_q = reward
                  
              # Update Q value for given state
              current_qs = current_qs_list[index]
              current_qs[action] = new_q
              
              X.append(current_state)
              Y.append(current_qs)
  </code></pre>
  <p>Since np.max(future_qs_list) should be np.max(future_qs_list[index]) since you're now getting the highest Q of the entire batch.
  Instead of the getting the highest Q from the current next state.</p>
  <p>It's like this after changing that (remember an epsilon of 1 means that you get 100% of your actions taken by the a dice roll so I let it go for a few more epochs, also tried it with the old code but indeed didn't get more then 50 steps (even after 400 epochs/episodes))</p>
  <pre><code>episode : 52, steps 16, epsilon : 0.7705488893118823
  episode : 53, steps 25, epsilon : 0.7666961448653229
  episode : 54, steps 25, epsilon : 0.7628626641409962
  episode : 55, steps 36, epsilon : 0.7590483508202912
  episode : 56, steps 32, epsilon : 0.7552531090661897
  episode : 57, steps 22, epsilon : 0.7514768435208588
  episode : 58, steps 55, epsilon : 0.7477194593032545
  episode : 59, steps 24, epsilon : 0.7439808620067382
  episode : 60, steps 46, epsilon : 0.7402609576967045
  episode : 61, steps 11, epsilon : 0.736559652908221
  episode : 62, steps 14, epsilon : 0.7328768546436799
  episode : 63, steps 13, epsilon : 0.7292124703704616
  episode : 64, steps 113, epsilon : 0.7255664080186093
  episode : 65, steps 33, epsilon : 0.7219385759785162
  episode : 66, steps 33, epsilon : 0.7183288830986236
  episode : 67, steps 39, epsilon : 0.7147372386831305
  episode : 68, steps 27, epsilon : 0.7111635524897149
  episode : 69, steps 22, epsilon : 0.7076077347272662
  episode : 70, steps 60, epsilon : 0.7040696960536299
  episode : 71, steps 40, epsilon : 0.7005493475733617
  episode : 72, steps 67, epsilon : 0.697046600835495
  episode : 73, steps 115, epsilon : 0.6935613678313175
  episode : 74, steps 61, epsilon : 0.6900935609921609
  episode : 75, steps 43, epsilon : 0.6866430931872001
  episode : 76, steps 21, epsilon : 0.6832098777212641
  episode : 77, steps 65, epsilon : 0.6797938283326578
  episode : 78, steps 45, epsilon : 0.6763948591909945
  episode : 79, steps 93, epsilon : 0.6730128848950395
  episode : 80, steps 200, epsilon : 0.6696478204705644
  episode : 81, steps 200, epsilon : 0.6662995813682115
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-15T22:54:23.440Z
Answers.LastActivityDate: 2020-08-15T23:26:19.590Z
Answers.LastEditDate: 2020-08-15T23:26:19.590Z
Answers.UserDisplayName: hal9000
Body: <p>I am implementing OpenAI gym's cartpole problem using Deep Q-Learning (DQN). I followed tutorials (video and otherwise) and learned all about it. I implemented a code for myself and I thought it should work, but the agent is not learning. I will really really really appreciate if someone can pinpoint where I am doing wrong.</p>
  <p>Note that I have a target neuaral network and a policy network already there. The code is as below.</p>
  <pre><code>import numpy as np
  import gym
  import random
  from keras.optimizers import Adam
  from keras.models import Sequential
  from keras.layers import Dense
  from collections import deque
  
  env = gym.make('CartPole-v0')
  
  EPISODES = 2000
  BATCH_SIZE = 32
  DISCOUNT = 0.95
  UPDATE_TARGET_EVERY = 5
  STATE_SIZE = env.observation_space.shape[0]
  ACTION_SIZE = env.action_space.n
  SHOW_EVERY = 50
  
  class DQNAgents:
      
      def __init__(self, state_size, action_size):
          self.state_size = state_size
          self.action_size = action_size
          self.replay_memory = deque(maxlen = 2000)
          self.gamma = 0.95
          self.epsilon = 1
          self.epsilon_decay = 0.995
          self.epsilon_min = 0.01
          self.model = self._build_model()
          self.target_model = self.model
          
          self.target_update_counter = 0
          print('Initialize the agent')
          
      def _build_model(self):
          model = Sequential()
          model.add(Dense(20, input_dim = self.state_size, activation = 'relu'))
          model.add(Dense(10, activation = 'relu'))
          model.add(Dense(self.action_size, activation = 'linear'))
          model.compile(loss = 'mse', optimizer = Adam(lr = 0.001))
          
          return model
  
      def update_replay_memory(self, current_state, action, reward, next_state, done):
          self.replay_memory.append((current_state, action, reward, next_state, done))
          
      def train(self, terminal_state):
          
          # Sample from replay memory
          minibatch = random.sample(self.replay_memory, BATCH_SIZE)
          
          #Picks the current states from the randomly selected minibatch
          current_states = np.array([t[0] for t in minibatch])
          current_qs_list= self.model.predict(current_states) #gives the Q value for the policy network
          new_state = np.array([t[3] for t in minibatch])
          future_qs_list = self.target_model.predict(new_state)
          
          X = []
          Y = []
          
          # This loop will run 32 times (actually minibatch times)
          for index, (current_state, action, reward, next_state, done) in enumerate(minibatch):
              
              if not done:
                  new_q = reward + DISCOUNT * np.max(future_qs_list)
              else:
                  new_q = reward
                  
              # Update Q value for given state
              current_qs = current_qs_list[index]
              current_qs[action] = new_q
              
              X.append(current_state)
              Y.append(current_qs)
          
          # Fitting the weights, i.e. reducing the loss using gradient descent
          self.model.fit(np.array(X), np.array(Y), batch_size = BATCH_SIZE, verbose = 0, shuffle = False)
          
         # Update target network counter every episode
          if terminal_state:
              self.target_update_counter += 1
              
          # If counter reaches set value, update target network with weights of main network
          if self.target_update_counter &gt; UPDATE_TARGET_EVERY:
              self.target_model.set_weights(self.model.get_weights())
              self.target_update_counter = 0
      
      def get_qs(self, state):
          return self.model.predict(np.array(state).reshape(-1, *state.shape))[0]
              
  
  ''' We start here'''
  
  agent = DQNAgents(STATE_SIZE, ACTION_SIZE)
  
  for e in range(EPISODES):
      
      done = False
      current_state = env.reset()
      time = 0 
      total_reward = 0
      while not done:
          if np.random.random() &gt; agent.epsilon:
              action = np.argmax(agent.get_qs(current_state))
          else:
              action = env.action_space.sample()
          
          next_state, reward, done, _ = env.step(action)
  
          agent.update_replay_memory(current_state, action, reward, next_state, done)
          
          if len(agent.replay_memory) &lt; BATCH_SIZE:
              pass
          else:
              agent.train(done)
              
          time+=1    
          current_state = next_state
          total_reward += reward
          
      print(f'episode : {e}, steps {time}, epsilon : {agent.epsilon}')
      
      if agent.epsilon &gt; agent.epsilon_min:
          agent.epsilon *= agent.epsilon_decay
  </code></pre>
  <p>Results for first 40ish iterations are below (look for the number of steps, they should be increasing and should reach a maximum of 199)</p>
  <pre><code>episode : 0, steps 14, epsilon : 1
  episode : 1, steps 13, epsilon : 0.995
  episode : 2, steps 17, epsilon : 0.990025
  episode : 3, steps 12, epsilon : 0.985074875
  episode : 4, steps 29, epsilon : 0.9801495006250001
  episode : 5, steps 14, epsilon : 0.9752487531218751
  episode : 6, steps 11, epsilon : 0.9703725093562657
  episode : 7, steps 13, epsilon : 0.9655206468094844
  episode : 8, steps 11, epsilon : 0.960693043575437
  episode : 9, steps 14, epsilon : 0.9558895783575597
  episode : 10, steps 39, epsilon : 0.9511101304657719
  episode : 11, steps 14, epsilon : 0.946354579813443
  episode : 12, steps 19, epsilon : 0.9416228069143757
  episode : 13, steps 16, epsilon : 0.9369146928798039
  episode : 14, steps 14, epsilon : 0.9322301194154049
  episode : 15, steps 18, epsilon : 0.9275689688183278
  episode : 16, steps 31, epsilon : 0.9229311239742362
  episode : 17, steps 14, epsilon : 0.918316468354365
  episode : 18, steps 21, epsilon : 0.9137248860125932
  episode : 19, steps 9, epsilon : 0.9091562615825302
  episode : 20, steps 26, epsilon : 0.9046104802746175
  episode : 21, steps 20, epsilon : 0.9000874278732445
  episode : 22, steps 53, epsilon : 0.8955869907338783
  episode : 23, steps 24, epsilon : 0.8911090557802088
  episode : 24, steps 14, epsilon : 0.8866535105013078
  episode : 25, steps 40, epsilon : 0.8822202429488013
  episode : 26, steps 10, epsilon : 0.8778091417340573
  episode : 27, steps 60, epsilon : 0.8734200960253871
  episode : 28, steps 17, epsilon : 0.8690529955452602
  episode : 29, steps 11, epsilon : 0.8647077305675338
  episode : 30, steps 42, epsilon : 0.8603841919146962
  episode : 31, steps 16, epsilon : 0.8560822709551227
  episode : 32, steps 12, epsilon : 0.851801859600347
  episode : 33, steps 12, epsilon : 0.8475428503023453
  episode : 34, steps 10, epsilon : 0.8433051360508336
  episode : 35, steps 30, epsilon : 0.8390886103705794
  episode : 36, steps 21, epsilon : 0.8348931673187264
  episode : 37, steps 24, epsilon : 0.8307187014821328
  episode : 38, steps 33, epsilon : 0.8265651079747222
  episode : 39, steps 32, epsilon : 0.8224322824348486
  episode : 40, steps 15, epsilon : 0.8183201210226743
  episode : 41, steps 20, epsilon : 0.8142285204175609
  episode : 42, steps 37, epsilon : 0.810157377815473
  episode : 43, steps 11, epsilon : 0.8061065909263957
  episode : 44, steps 30, epsilon : 0.8020760579717637
  episode : 45, steps 11, epsilon : 0.798065677681905
  episode : 46, steps 34, epsilon : 0.7940753492934954
  episode : 47, steps 12, epsilon : 0.7901049725470279
  episode : 48, steps 26, epsilon : 0.7861544476842928
  episode : 49, steps 19, epsilon : 0.7822236754458713
  episode : 50, steps 20, epsilon : 0.778312557068642
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-11T15:49:12.063Z
Comments.Text: I have also tried by setting the epsilon (parameter for epsilon-greedy) to 0.1 from the beginning. The learning still remain worse than what it is when epsilon starts from 1. Apparently there is some issue in updating the best Q value for the selected action. But I cannot find what exactly I am missing, doing wrong, or what I should add.
Comments.UserDisplayName: SJa
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-15T22:01:51.037Z
Comments.Text: You know that your epsilon needs to be high in the beginning since it's the way you search though your search space... (eploitation vs exploration)
Comments.UserDisplayName: hal9000
Tags: <reinforcement-learning><dqn><deep-neural-networks><open-ai><gym>
Title: My Deep Q-Learning Network does not learn for OpenAI gym's cartpole problem
UserDisplayName: SJa
ThreadText: # My Deep Q-Learning Network does not learn for OpenAI gym's cartpole problem
  
  (Asked by: username_0 on 2020-08-11)
  
  I am implementing OpenAI gym's cartpole problem using Deep Q-Learning (DQN). I followed tutorials (video and otherwise) and learned all about it. I implemented a code for myself and I thought it should work, but the agent is not learning. I will really really really appreciate if someone can pinpoint where I am doing wrong.
  
  Note that I have a target neuaral network and a policy network already there. The code is as below.
  
  ```
  import numpy as np
  import gym
  import random
  from keras.optimizers import Adam
  from keras.models import Sequential
  from keras.layers import Dense
  from collections import deque
  
  env = gym.make('CartPole-v0')
  
  EPISODES = 2000
  BATCH_SIZE = 32
  DISCOUNT = 0.95
  UPDATE_TARGET_EVERY = 5
  STATE_SIZE = env.observation_space.shape[0]
  ACTION_SIZE = env.action_space.n
  SHOW_EVERY = 50
  
  class DQNAgents:
      
      def __init__(self, state_size, action_size):
          self.state_size = state_size
          self.action_size = action_size
          self.replay_memory = deque(maxlen = 2000)
          self.gamma = 0.95
          self.epsilon = 1
          self.epsilon_decay = 0.995
          self.epsilon_min = 0.01
          self.model = self._build_model()
          self.target_model = self.model
          
          self.target_update_counter = 0
          print('Initialize the agent')
          
      def _build_model(self):
          model = Sequential()
          model.add(Dense(20, input_dim = self.state_size, activation = 'relu'))
          model.add(Dense(10, activation = 'relu'))
          model.add(Dense(self.action_size, activation = 'linear'))
          model.compile(loss = 'mse', optimizer = Adam(lr = 0.001))
          
          return model
  
      def update_replay_memory(self, current_state, action, reward, next_state, done):
          self.replay_memory.append((current_state, action, reward, next_state, done))
          
      def train(self, terminal_state):
          
          # Sample from replay memory
          minibatch = random.sample(self.replay_memory, BATCH_SIZE)
          
          #Picks the current states from the randomly selected minibatch
          current_states = np.array([t[0] for t in minibatch])
          current_qs_list= self.model.predict(current_states) #gives the Q value for the policy network
          new_state = np.array([t[3] for t in minibatch])
          future_qs_list = self.target_model.predict(new_state)
          
          X = []
          Y = []
          
          # This loop will run 32 times (actually minibatch times)
          for index, (current_state, action, reward, next_state, done) in enumerate(minibatch):
              
              if not done:
                  new_q = reward + DISCOUNT * np.max(future_qs_list)
              else:
                  new_q = reward
                  
              # Update Q value for given state
              current_qs = current_qs_list[index]
              current_qs[action] = new_q
              
              X.append(current_state)
              Y.append(current_qs)
          
          # Fitting the weights, i.e. reducing the loss using gradient descent
          self.model.fit(np.array(X), np.array(Y), batch_size = BATCH_SIZE, verbose = 0, shuffle = False)
          
         # Update target network counter every episode
          if terminal_state:
              self.target_update_counter += 1
              
          # If counter reaches set value, update target network with weights of main network
          if self.target_update_counter > UPDATE_TARGET_EVERY:
              self.target_model.set_weights(self.model.get_weights())
              self.target_update_counter = 0
      
      def get_qs(self, state):
          return self.model.predict(np.array(state).reshape(-1, *state.shape))[0]
              
  
  ''' We start here'''
  
  agent = DQNAgents(STATE_SIZE, ACTION_SIZE)
  
  for e in range(EPISODES):
      
      done = False
      current_state = env.reset()
      time = 0 
      total_reward = 0
      while not done:
          if np.random.random() > agent.epsilon:
              action = np.argmax(agent.get_qs(current_state))
          else:
              action = env.action_space.sample()
          
          next_state, reward, done, _ = env.step(action)
  
          agent.update_replay_memory(current_state, action, reward, next_state, done)
          
          if len(agent.replay_memory) < BATCH_SIZE:
              pass
          else:
              agent.train(done)
              
          time+=1    
          current_state = next_state
          total_reward += reward
          
      print(f'episode : {e}, steps {time}, epsilon : {agent.epsilon}')
      
      if agent.epsilon > agent.epsilon_min:
          agent.epsilon *= agent.epsilon_decay
  
  ```
  
  Results for first 40ish iterations are below (look for the number of steps, they should be increasing and should reach a maximum of 199)
  
  ```
  episode : 0, steps 14, epsilon : 1
  episode : 1, steps 13, epsilon : 0.995
  episode : 2, steps 17, epsilon : 0.990025
  episode : 3, steps 12, epsilon : 0.985074875
  episode : 4, steps 29, epsilon : 0.9801495006250001
  episode : 5, steps 14, epsilon : 0.9752487531218751
  episode : 6, steps 11, epsilon : 0.9703725093562657
  episode : 7, steps 13, epsilon : 0.9655206468094844
  episode : 8, steps 11, epsilon : 0.960693043575437
  episode : 9, steps 14, epsilon : 0.9558895783575597
  episode : 10, steps 39, epsilon : 0.9511101304657719
  episode : 11, steps 14, epsilon : 0.946354579813443
  episode : 12, steps 19, epsilon : 0.9416228069143757
  episode : 13, steps 16, epsilon : 0.9369146928798039
  episode : 14, steps 14, epsilon : 0.9322301194154049
  episode : 15, steps 18, epsilon : 0.9275689688183278
  episode : 16, steps 31, epsilon : 0.9229311239742362
  episode : 17, steps 14, epsilon : 0.918316468354365
  episode : 18, steps 21, epsilon : 0.9137248860125932
  episode : 19, steps 9, epsilon : 0.9091562615825302
  episode : 20, steps 26, epsilon : 0.9046104802746175
  episode : 21, steps 20, epsilon : 0.9000874278732445
  episode : 22, steps 53, epsilon : 0.8955869907338783
  episode : 23, steps 24, epsilon : 0.8911090557802088
  episode : 24, steps 14, epsilon : 0.8866535105013078
  episode : 25, steps 40, epsilon : 0.8822202429488013
  episode : 26, steps 10, epsilon : 0.8778091417340573
  episode : 27, steps 60, epsilon : 0.8734200960253871
  episode : 28, steps 17, epsilon : 0.8690529955452602
  episode : 29, steps 11, epsilon : 0.8647077305675338
  episode : 30, steps 42, epsilon : 0.8603841919146962
  episode : 31, steps 16, epsilon : 0.8560822709551227
  episode : 32, steps 12, epsilon : 0.851801859600347
  episode : 33, steps 12, epsilon : 0.8475428503023453
  episode : 34, steps 10, epsilon : 0.8433051360508336
  episode : 35, steps 30, epsilon : 0.8390886103705794
  episode : 36, steps 21, epsilon : 0.8348931673187264
  episode : 37, steps 24, epsilon : 0.8307187014821328
  episode : 38, steps 33, epsilon : 0.8265651079747222
  episode : 39, steps 32, epsilon : 0.8224322824348486
  episode : 40, steps 15, epsilon : 0.8183201210226743
  episode : 41, steps 20, epsilon : 0.8142285204175609
  episode : 42, steps 37, epsilon : 0.810157377815473
  episode : 43, steps 11, epsilon : 0.8061065909263957
  episode : 44, steps 30, epsilon : 0.8020760579717637
  episode : 45, steps 11, epsilon : 0.798065677681905
  episode : 46, steps 34, epsilon : 0.7940753492934954
  episode : 47, steps 12, epsilon : 0.7901049725470279
  episode : 48, steps 26, epsilon : 0.7861544476842928
  episode : 49, steps 19, epsilon : 0.7822236754458713
  episode : 50, steps 20, epsilon : 0.778312557068642
  
  ```
  
  **Tags:** reinforcement-learning, dqn, deep-neural-networks, open-ai, gym
  
  > **Comments:**
  > **username_0**: I have also tried by setting the epsilon (parameter for epsilon-greedy) to 0.1 from the beginning. The learning still remain worse than what it is when epsilon starts from 1. Apparently there is some issue in updating the best Q value for the selected action. But I cannot find what exactly I am missing, doing wrong, or what I should add.  
  > **username_1**: You know that your epsilon needs to be high in the beginning since it's the way you search though your search space... (eploitation vs exploration)  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-08-15. Score: 3 | Accepted answer
  
  There is a really small mistake in here that causes the problem:
  
  ```
  
  for index, (current_state, action, reward, next_state, done) in enumerate(minibatch):
              if not done:
                  new_q = reward + DISCOUNT * np.max(future_qs_list) #HERE 
              else:
                  new_q = reward
                  
              # Update Q value for given state
              current_qs = current_qs_list[index]
              current_qs[action] = new_q
              
              X.append(current_state)
              Y.append(current_qs)
  
  ```
  
  Since np.max(future\_qs\_list) should be np.max(future\_qs\_list[index]) since you're now getting the highest Q of the entire batch.
  Instead of the getting the highest Q from the current next state.
  
  It's like this after changing that (remember an epsilon of 1 means that you get 100% of your actions taken by the a dice roll so I let it go for a few more epochs, also tried it with the old code but indeed didn't get more then 50 steps (even after 400 epochs/episodes))
  
  ```
  episode : 52, steps 16, epsilon : 0.7705488893118823
  episode : 53, steps 25, epsilon : 0.7666961448653229
  episode : 54, steps 25, epsilon : 0.7628626641409962
  episode : 55, steps 36, epsilon : 0.7590483508202912
  episode : 56, steps 32, epsilon : 0.7552531090661897
  episode : 57, steps 22, epsilon : 0.7514768435208588
  episode : 58, steps 55, epsilon : 0.7477194593032545
  episode : 59, steps 24, epsilon : 0.7439808620067382
  episode : 60, steps 46, epsilon : 0.7402609576967045
  episode : 61, steps 11, epsilon : 0.736559652908221
  episode : 62, steps 14, epsilon : 0.7328768546436799
  episode : 63, steps 13, epsilon : 0.7292124703704616
  episode : 64, steps 113, epsilon : 0.7255664080186093
  episode : 65, steps 33, epsilon : 0.7219385759785162
  episode : 66, steps 33, epsilon : 0.7183288830986236
  episode : 67, steps 39, epsilon : 0.7147372386831305
  episode : 68, steps 27, epsilon : 0.7111635524897149
  episode : 69, steps 22, epsilon : 0.7076077347272662
  episode : 70, steps 60, epsilon : 0.7040696960536299
  episode : 71, steps 40, epsilon : 0.7005493475733617
  episode : 72, steps 67, epsilon : 0.697046600835495
  episode : 73, steps 115, epsilon : 0.6935613678313175
  episode : 74, steps 61, epsilon : 0.6900935609921609
  episode : 75, steps 43, epsilon : 0.6866430931872001
  episode : 76, steps 21, epsilon : 0.6832098777212641
  episode : 77, steps 65, epsilon : 0.6797938283326578
  episode : 78, steps 45, epsilon : 0.6763948591909945
  episode : 79, steps 93, epsilon : 0.6730128848950395
  episode : 80, steps 200, epsilon : 0.6696478204705644
  episode : 81, steps 200, epsilon : 0.6662995813682115
  
  ```
  
  
  ---
  
  ### Answer by username_2 on 2020-08-15. Score: 0
  
  I think the problem is with openAI gym CartPole-v0 environment reward structure.
  The reward is always +1 for each time step. So if pole falls reward is +1 itself.
  So we need to check and redefine the reward for this case.
  So in the train function try this:
  
  ```
  if not done:
      new_q = reward + DISCOUNT * np.max(future_qs_list)
  else:
      # if done assign some negative reward
      new_q = -20
  
  ```
  
  (Or change the reward during replay buffer update)
  
  Check the lines 81 and 82 in Qlearning.py code in [this](https://github.com/girishdhegde/reinforcement-learning/) repo for further clarification.
  
  > **Comments:**
  > **username_1**: You really don't need to edit the reward structure. The reward structure is fine. Such a high negative reward can even have negative effects.  
  > **username_3**: @hal I’m not sure that that’s true (that you don’t need to change it). If the environment gives +1 on termination then that can be problematic for the agent, it is better to change it to 0 or -1.  
  > **username_1**: yeah, I have to agree on that, 0 or -1 seems reasonable to me too but the reward structure is fine enough to be able to solve without doing that.  
  > **username_2**: @hal I experimented with different reward stuctures for cartpole using classical Q-learning(Not DQN). I found that reward between -10 to -20 for terminal state working better. My experimentation code and graph can be found [here](https://github.com/girishdhegde/reinforcement-learning/tree/master). But I found [this](https://gym.openai.com/evaluations/eval_aHf1Kmc4QIKm5oPcJJToBA/) code having reward of -200 for terminal state in official gym website.  
  
  ---
  
  