Answers.Body: <p>I see some issues in your code of the environment. Firstly, and probably most importantly, you should not be incrementing the reward. In your code, every time the agent hits <span class="math-container">$t=475$</span> for example, the reward given by the environment increases by 1. So if the agent oscillates between <span class="math-container">$t=450$</span> and <span class="math-container">$t=475$</span>, at each timestep the environment gives a greater and greater reward. Your Q values have no reason to converge. Instead of incrementing the reward, you should just return a reward of <span class="math-container">$1$</span> at those two desired states and return <span class="math-container">$0$</span> otherwise.</p>
  <p>Secondly, it appears that you don't reset the state during <code>reset()</code>, I think you should have <code>self.time = 0</code> in there (unless this was intentional).</p>
  <p><strong>Edit</strong> : I also question why you're using DQN for this problem. This problem should be very easy to solve with standard (tabular) Q Learning.</p>
  <p><strong>Edit 2</strong> : Your training loop looks wrong. You reset the environment at each iteration in your loop, so your episodes are one timestep long each. What I think you want instead is to have another loop under <code>reset()</code> something like</p>
  <pre><code>for episode in range(episodes):
    s = env.reset()
    episode_score = 0 
    for step in range(max_timesteps):
      a = agent.predict_action(s)
      ns, r, d, _ = env.step(a)
      episode_score += r
     ... 
      agent.update_policy()
     ... 
    scores.append(episode_score)
   ... 
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-11T15:31:18.333Z
Answers.Text: thanks, I just corrected the typo in the reset function and changed the reward scheme.. unfortunately, still no convergence towards t = 475 or 450 ... and to answer your question: i am still new to reinforcement learning, and wanted to learn how to implement DQN  on a simple problem, just for the sake of learning and understanding
Answers.UserDisplayName: relaxon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-11T17:44:55.650Z
Answers.Text: I also suggest you increase gamma. Gamma=0.5 is fairly low, and it's very possible that the rewards are just not propagating back to other states due to the exponential decay. Try with gamma=0.99 or something. Not sure if this will be the definitive solution, but it can't hurt.
Answers.UserDisplayName: harwiltz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-11T17:46:30.987Z
Answers.Text: Also, can you describe the results you get? As in, what do you mean by "no convergence towards t=..."? Can you talk about what you expect vs what you observe?
Answers.UserDisplayName: harwiltz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-11T19:35:34.107Z
Answers.Text: thank you, so let me explain: in the end i want the agent to do the following 475 +0 = 475 --> 475 + 0 = 475 --> 475 +0 = 475.... OR something like 475 -25 = 450 --> 450 +25 = 475 --> 475 -25 = 450 .... (i want the sum to always be 475 or 450)... what I observe: it just takes the same action for any state ( for example 450 +25 = 475 --> 475 + 25 = 500 -->  525 + 25 = 550 --> 550 + 25 = 575....).. so it just learns to take always the same action over and over again without caring about the result
Answers.UserDisplayName: relaxon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-11T21:05:15.607Z
Answers.Text: Noticed another issue, updated my original answer.
Answers.UserDisplayName: harwiltz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-12T08:18:14.947Z
Answers.Text: Thank you very much for your comments and suggestions.. unfortunately i still don't get what I expect the agent to learn ( i changed the hyperparameters too, but no success so far)... probably DQN is not very well suited for this type of task as you mentioned
Answers.UserDisplayName: relaxon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-12T12:54:00.593Z
Answers.Text: If you're just doing this as a learning exercise, why don't you try an openai gym environment? DQN definitely can solve cartpole, so I suggest you try that to figure out if there's something wrong with your DQN implementation.
Answers.UserDisplayName: harwiltz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-11T15:16:25.443Z
Answers.LastActivityDate: 2020-10-11T21:04:59.300Z
Answers.LastEditDate: 2020-10-11T21:04:59.300Z
Answers.UserDisplayName: harwiltz
Body: <p>I am pretty new to RL and I am trying to code a simple RL task with pytorch.</p>
  <p>The goal/task is the following:
  The initial state is <span class="math-container">$t_o$</span> and the agent takes an action <span class="math-container">$\Delta_t$</span>: <span class="math-container">$t_o +\Delta_t = t_1$</span>.</p>
  <p>If <span class="math-container">$t_1$</span> equals 450 or 475 then it gets a reward, else he does not get a reward.</p>
  <p>I am training the agent with DQN algorithm on a NN ( with: 2 Linear layes: fist layer n_in=1 n_out 128 and second layer n_in=128 and n_out=5):</p>
  <blockquote>
  <p>observation space(<span class="math-container">$t_i$</span>) is 700 --&gt; <span class="math-container">$t_i \in [0,700[$</span><br />
  action space (<span class="math-container">$\Delta_t$</span>) is 5 --&gt; (<span class="math-container">$\Delta_t \in [-50,-25,0,25,50]$</span>)</p>
  </blockquote>
  <blockquote>
  <p>epsilon_start=0.9#e-greedy threshold start value<br />
  epsilon_end=0.01#e-greedy threshold end value<br />
  epsilon_decay=200#e-greedy threshold decay learning_rate=0.001# NN<br />
  optimizer learning rate batch_size=64#Q-learning batch size</p>
  </blockquote>
  <p>Unfortunately it does not seem to converge to the values <span class="math-container">$t_i=$</span> 450 or 475. I doesn't seem to care about getting a reward.
  How can I improve my code so that the agent learns what I am trying to teach him?
  I put my code below in case the explanations were not clear enough:</p>
  <pre><code>
  import gym
  from gym import spaces
  
  class RL_env(gym.Env):
      metadata = {'render.modes': ['human']}
  
      
      def __init__(self):
          super(RL_env, self).__init__()
          
          n_actions_delta = 1 #delta_t
          self.action_space = spaces.Discrete(5)
          
          n_observations = 1 #time
      
          self.observation_space = spaces.Discrete(700)
         
          #initial time
          self.time = 0
          
          self.done = 0
          self.reward = 0
  
      def reset(self):
          self.reward = 0
          self.done = False
          return self.reward
         
      def step(self,delta_t):
          print('self time',self.time)
          d_t = np.arange(-50,70,25)
          
          self.time = (self.time + d_t[delta_t])%700
          print('delta time',d_t[delta_t],'--&gt;','self time',self.time)
          
         
          
          if self.time == 475 or self.time == 450:
              self.reward = 1
              
              
          else:
              self.reward += 0
          
              
          info = {}
          print('total reward',self.reward)
          print('\n')
          return self.time,self.reward, self.done, info
      
  
      
      
      def render(self, mode='human', close=False):
          print()
  </code></pre>
  <pre><code>import gym
  import numpy as np
  import matplotlib.pyplot as plt
  import torch
  import torch.nn as nn
  import torch.optim as optim
  import torch.nn.functional as F
  from torch.autograd import Variable
  from torch.distributions import Categorical
  dtype = torch.float
  device = torch.device(&quot;cpu&quot;)
  import random
  import math
  import sys
  if not sys.warnoptions:#igrnore warnings
      import warnings
      warnings.simplefilter(&quot;ignore&quot;)
  
  #hyper parameters
  epsilon_start=0.9
  #e-greedy threshold start value
  epsilon_end=0.01#e-greedy threshold end value
  epsilon_decay=200#e-greedy threshold decay
  learning_rate=0.001# NN optimizer learning rate
  batch_size=64#Q-learning batch size 
  
  env = RL_env()
  
  
  #use replay memory (-&gt; to stabilize and improve our algorithm)for training: store transitions observed by agent,
  #then reuse this data later
  #sample from it randomly (batch built by transitions are decorrelated)
  class ReplayMemory:#allowes the agent to learn from earlier memories (speed up learning and break undesirable temporal correlations)
      def __init__(self, capacity):
          self.capacity = capacity
          self.memory = []
      def push(self, transition):#saves transition
          self.memory.append(transition)
          if len(self.memory)&gt;self.capacity:#if length of memory arra is larger than capacity (fixed)
              del self.memory[0]#remove 0th element
  
      def sample(self, batch_number):#samples randomly a transition to build batch
          return random.sample(self.memory, batch_number)
  
      def __len__(self):
          return len(self.memory)
      
  #Dqn NN (we want to maximize the discounted, cumulative reward)
  #idea of Q-learning: we want to approximate with NN maximal Q-function (gives max return of action in given state)
  #training update rule: use the fact that every Q-function for some policy obeys the Bellman equation
  #difference between the two sides of the equality is known as the temporal difference error (we want to min -&gt; Huber loss)
  #calculate over batch of transitions sampled from the replay memory
  class DqnNet(nn.Module):
      def __init__(self):
          super(DqnNet, self).__init__()
          
          state_space = 1
          action_space = env.action_space.n
          num_hid = 128
          self.fc1 = nn.Linear(state_space, num_hid)
          self.fc2 = nn.Linear(num_hid, action_space)
          self.gamma=0.5 #Q-learning discount factor (ensures that reward sum converges, 
                          #makes actions from far future less important)
      def forward(self, x):
          x = F.relu(self.fc1(x))
          x = F.sigmoid(self.fc2(x))
          return x
  
  #select action accordingly to epsilon greedy policy
  #sometimes we use model for choosing action, other times sample uniformly 
  #probability of choosing a random action will start at epsilon_start and will decay (epsilon_decay) exponentially
  #towards epsilon_end
  steps_done=0
  def predict_action(state):
      global steps_done
      sample=random.random()#random number
      eps_threshold=epsilon_end+(epsilon_start-epsilon_end)*math.exp(-1.*steps_done/epsilon_decay)
      steps_done += 1
      if sample&gt;eps_threshold:  
          x  = eps_threshold,model(Variable(state,).type(torch.FloatTensor)).data.max(0)[1].view(1, 1)
          return x#chose action from model
      
      else:
          x = eps_threshold,torch.tensor([[random.randrange(env.action_space.n)]])
          return x#choose random action uniformly
  
  #wtih the update_policy function we perform a single step of the optimization
  #first sample a batch, concatenate all the tensors into a single one, compute Q-value and max Q-value, 
  #and combine them into loss
  def update_policy():
      if len(memory)&lt;batch_size:#we want to sample a batch of size 64
          return
      transitions = memory.sample(batch_size)#take random transition batch from experience replay memory
      batch_state, batch_action, batch_next_state, batch_reward = zip(*transitions)#convert batch-array of Transitions
                                                                                #to Transition of batch-arrays   
      #--&gt;zip(*) takes iterables as arguments and return iterator
      
      batch_state = Variable(torch.cat(batch_state))#concatenate given sequence tensors in the given dimension
      batch_state = batch_state.resize(batch_size,1)
      batch_action = Variable(torch.cat(batch_action))
      batch_next_state = Variable(torch.cat(batch_next_state))
      batch_next_state = batch_next_state.reshape(batch_size,1)
      batch_reward = Variable(torch.cat(batch_reward))
      
      #print('model batch state',model(Variable(batch_state[0])))
      current_q_values = model(batch_state).gather(1, batch_action)#current Q-values estimated for all actions,
                                                                   #compute Q, then select the columns of actions taken,
                                                                   #these are the actions which would've been taken
                                                                   #for each batch state according to policy_net
      max_next_q_values = model(batch_next_state).detach().max(1)[0]#predicted Q-values for non-final-next-states
                                                                    #(-&gt; gives max Q)
      expected_q_values = batch_reward + (model.gamma * max_next_q_values)
  
      #loss is measured from error between current and newly expected Q values (Huber Loss)
      loss = F.smooth_l1_loss(current_q_values, expected_q_values)
  
      # backpropagation of loss to NN --&gt; optimize model
      optimizer.zero_grad()
      loss.backward()
      optimizer.step()
      return loss, np.sum(expected_q_values.numpy())
      
  
  def train(episodes):
      scores = []
      Losses = []
      Bellman = []
      Epsilon = []
      Times = []
      Deltas = []
      
      
      
      for episode in range(episodes):  
          state=env.reset()#reset environment
          print('\n')
          print('episode',episode)
             
          epsilon_action = predict_action(torch.FloatTensor([state]))
          
          action = epsilon_action[1] #after each time step predict action
  
          next_state, reward, done,info = env.step(action.item())#step through environment using chosen action
          
          epsilon = epsilon_action[0]
          Epsilon.append(epsilon)
          print(reward,'reward')
                
          state=next_state
          Times.append(state)
          scores.append(reward)            
  
      
          memory.push((torch.FloatTensor([state]),action,torch.FloatTensor([next_state]),
                           torch.FloatTensor([reward])))#action is already a tensor
          up = update_policy()#update_policy()#update policy
              
          if up != None:
              Losses.append(Variable(up[0]))
              print('loss',Variable(up[0]))
              Bellman.append(up[1])
  
          #calculate score to determine when the environment has been solved
          mean_score=np.mean(scores[-50:])#mean of score of last 50 episodes
          #every 50th episode print score
          if episode%50 == 0:
              print('Episode {}\tScore: {}\tAverage score(last 50 episodes): {:.2f}'.format(episode,scores[-50:],mean_score))
  
      
      #print('Losses',Losses)
      Losses = torch.stack(Losses).numpy()
      #print('Losses',Losses)
      plt.plot(np.arange(len(Losses)),Losses)
      plt.xlabel('Training iterations')
      plt.ylabel('Loss')
      plt.show()
      
      Bellman = np.array(Bellman)
      #print('Bellman',Bellman,'\n')
      plt.plot(np.arange(len(Bellman)),Bellman)
      plt.xlabel('Training iterations')
      plt.ylabel('Bellman target')
      plt.show()
      
      #print('scores',scores)
      plt.plot(np.arange(len(scores)),scores)
      plt.xlabel('Training iterations')
      plt.ylabel('Reward')
      plt.show()
      
      #print('epsilon',Epsilon)
      plt.plot(np.arange(len(Epsilon)),Epsilon)
      plt.xlabel('Training iterations')
      plt.ylabel('Epsilon')
      plt.show()
      
      print('Times',Times[-25:])
      print('Deltas',Deltas[-25:])
      
      Times = np.array(Times)
      print('Times',Times)
      #plt.figure(figsize=(31,20))
      plt.figure(figsize=(9,7))
      plt.plot(np.arange(len(Times)),(np.array(Times)))
      plt.xlabel('Training iterations')
      plt.ylabel('t')
      plt.show()
       
      Times_1 = np.array(Times[-300:])
      print('t',Times)
      plt.figure(figsize=(9,7))
      plt.plot(np.arange(len(Times_1)),(np.array(Times_1)))
      plt.xlabel('Last 300 Training iterations')
      plt.ylabel('t')
      plt.ylim(0,1000)
      plt.show()
      
  model = DqnNet()#policy         
  memory = ReplayMemory(20000)
  optimizer = optim.Adam(model.parameters(), lr=learning_rate)
  
  train(10000)
  </code></pre>
  
Tags: <reinforcement-learning><q-learning><dqn>
Title: Reinforcement learning simple problem: agent not learning, wrong action
UserDisplayName: relaxon
ThreadText: # Reinforcement learning simple problem: agent not learning, wrong action
  
  (Asked by: username_0 on 2020-10-11)
  
  I am pretty new to RL and I am trying to code a simple RL task with pytorch.
  
  The goal/task is the following:
  The initial state is $t\_o$ and the agent takes an action $\Delta\_t$: $t\_o +\Delta\_t = t\_1$.
  
  If $t\_1$ equals 450 or 475 then it gets a reward, else he does not get a reward.
  
  I am training the agent with DQN algorithm on a NN ( with: 2 Linear layes: fist layer n\_in=1 n\_out 128 and second layer n\_in=128 and n\_out=5):
  
  > observation space($t\_i$) is 700 --> $t\_i \in [0,700[$  
  > action space ($\Delta\_t$) is 5 --> ($\Delta\_t \in [-50,-25,0,25,50]$)
  
  > epsilon\_start=0.9#e-greedy threshold start value  
  > epsilon\_end=0.01#e-greedy threshold end value  
  > epsilon\_decay=200#e-greedy threshold decay learning\_rate=0.001# NN  
  > optimizer learning rate batch\_size=64#Q-learning batch size
  
  Unfortunately it does not seem to converge to the values $t\_i=$ 450 or 475. I doesn't seem to care about getting a reward.
  How can I improve my code so that the agent learns what I am trying to teach him?
  I put my code below in case the explanations were not clear enough:
  
  ```
  
  import gym
  from gym import spaces
  
  class RL_env(gym.Env):
      metadata = {'render.modes': ['human']}
  
      
      def __init__(self):
          super(RL_env, self).__init__()
          
          n_actions_delta = 1 #delta_t
          self.action_space = spaces.Discrete(5)
          
          n_observations = 1 #time
      
          self.observation_space = spaces.Discrete(700)
         
          #initial time
          self.time = 0
          
          self.done = 0
          self.reward = 0
  
      def reset(self):
          self.reward = 0
          self.done = False
          return self.reward
         
      def step(self,delta_t):
          print('self time',self.time)
          d_t = np.arange(-50,70,25)
          
          self.time = (self.time + d_t[delta_t])%700
          print('delta time',d_t[delta_t],'-->','self time',self.time)
          
         
          
          if self.time == 475 or self.time == 450:
              self.reward = 1
              
              
          else:
              self.reward += 0
          
              
          info = {}
          print('total reward',self.reward)
          print('\n')
          return self.time,self.reward, self.done, info
      
  
      
      
      def render(self, mode='human', close=False):
          print()
  
  ```
  
  ```
  import gym
  import numpy as np
  import matplotlib.pyplot as plt
  import torch
  import torch.nn as nn
  import torch.optim as optim
  import torch.nn.functional as F
  from torch.autograd import Variable
  from torch.distributions import Categorical
  dtype = torch.float
  device = torch.device("cpu")
  import random
  import math
  import sys
  if not sys.warnoptions:#igrnore warnings
      import warnings
      warnings.simplefilter("ignore")
  
  #hyper parameters
  epsilon_start=0.9
  #e-greedy threshold start value
  epsilon_end=0.01#e-greedy threshold end value
  epsilon_decay=200#e-greedy threshold decay
  learning_rate=0.001# NN optimizer learning rate
  batch_size=64#Q-learning batch size 
  
  env = RL_env()
  
  #use replay memory (-> to stabilize and improve our algorithm)for training: store transitions observed by agent,
  #then reuse this data later
  #sample from it randomly (batch built by transitions are decorrelated)
  class ReplayMemory:#allowes the agent to learn from earlier memories (speed up learning and break undesirable temporal correlations)
      def __init__(self, capacity):
          self.capacity = capacity
          self.memory = []
      def push(self, transition):#saves transition
          self.memory.append(transition)
          if len(self.memory)>self.capacity:#if length of memory arra is larger than capacity (fixed)
              del self.memory[0]#remove 0th element
  
      def sample(self, batch_number):#samples randomly a transition to build batch
          return random.sample(self.memory, batch_number)
  
      def __len__(self):
          return len(self.memory)
      
  #Dqn NN (we want to maximize the discounted, cumulative reward)
  #idea of Q-learning: we want to approximate with NN maximal Q-function (gives max return of action in given state)
  #training update rule: use the fact that every Q-function for some policy obeys the Bellman equation
  #difference between the two sides of the equality is known as the temporal difference error (we want to min -> Huber loss)
  #calculate over batch of transitions sampled from the replay memory
  class DqnNet(nn.Module):
      def __init__(self):
          super(DqnNet, self).__init__()
          
          state_space = 1
          action_space = env.action_space.n
          num_hid = 128
          self.fc1 = nn.Linear(state_space, num_hid)
          self.fc2 = nn.Linear(num_hid, action_space)
          self.gamma=0.5 #Q-learning discount factor (ensures that reward sum converges, 
                          #makes actions from far future less important)
      def forward(self, x):
          x = F.relu(self.fc1(x))
          x = F.sigmoid(self.fc2(x))
          return x
  
  #select action accordingly to epsilon greedy policy
  #sometimes we use model for choosing action, other times sample uniformly 
  #probability of choosing a random action will start at epsilon_start and will decay (epsilon_decay) exponentially
  #towards epsilon_end
  steps_done=0
  def predict_action(state):
      global steps_done
      sample=random.random()#random number
      eps_threshold=epsilon_end+(epsilon_start-epsilon_end)*math.exp(-1.*steps_done/epsilon_decay)
      steps_done += 1
      if sample>eps_threshold:  
          x  = eps_threshold,model(Variable(state,).type(torch.FloatTensor)).data.max(0)[1].view(1, 1)
          return x#chose action from model
      
      else:
          x = eps_threshold,torch.tensor([[random.randrange(env.action_space.n)]])
          return x#choose random action uniformly
  
  #wtih the update_policy function we perform a single step of the optimization
  #first sample a batch, concatenate all the tensors into a single one, compute Q-value and max Q-value, 
  #and combine them into loss
  def update_policy():
      if len(memory)<batch_size:#we want to sample a batch of size 64
          return
      transitions = memory.sample(batch_size)#take random transition batch from experience replay memory
      batch_state, batch_action, batch_next_state, batch_reward = zip(*transitions)#convert batch-array of Transitions
                                                                                #to Transition of batch-arrays   
      #-->zip(*) takes iterables as arguments and return iterator
      
      batch_state = Variable(torch.cat(batch_state))#concatenate given sequence tensors in the given dimension
      batch_state = batch_state.resize(batch_size,1)
      batch_action = Variable(torch.cat(batch_action))
      batch_next_state = Variable(torch.cat(batch_next_state))
      batch_next_state = batch_next_state.reshape(batch_size,1)
      batch_reward = Variable(torch.cat(batch_reward))
      
      #print('model batch state',model(Variable(batch_state[0])))
      current_q_values = model(batch_state).gather(1, batch_action)#current Q-values estimated for all actions,
                                                                   #compute Q, then select the columns of actions taken,
                                                                   #these are the actions which would've been taken
                                                                   #for each batch state according to policy_net
      max_next_q_values = model(batch_next_state).detach().max(1)[0]#predicted Q-values for non-final-next-states
                                                                    #(-> gives max Q)
      expected_q_values = batch_reward + (model.gamma * max_next_q_values)
  
      #loss is measured from error between current and newly expected Q values (Huber Loss)
      loss = F.smooth_l1_loss(current_q_values, expected_q_values)
  
      # backpropagation of loss to NN --> optimize model
      optimizer.zero_grad()
      loss.backward()
      optimizer.step()
      return loss, np.sum(expected_q_values.numpy())
      
  
  def train(episodes):
      scores = []
      Losses = []
      Bellman = []
      Epsilon = []
      Times = []
      Deltas = []
      
      
      
      for episode in range(episodes):  
          state=env.reset()#reset environment
          print('\n')
          print('episode',episode)
             
          epsilon_action = predict_action(torch.FloatTensor([state]))
          
          action = epsilon_action[1] #after each time step predict action
  
          next_state, reward, done,info = env.step(action.item())#step through environment using chosen action
          
          epsilon = epsilon_action[0]
          Epsilon.append(epsilon)
          print(reward,'reward')
                
          state=next_state
          Times.append(state)
          scores.append(reward)            
  
      
          memory.push((torch.FloatTensor([state]),action,torch.FloatTensor([next_state]),
                           torch.FloatTensor([reward])))#action is already a tensor
          up = update_policy()#update_policy()#update policy
              
          if up != None:
              Losses.append(Variable(up[0]))
              print('loss',Variable(up[0]))
              Bellman.append(up[1])
  
          #calculate score to determine when the environment has been solved
          mean_score=np.mean(scores[-50:])#mean of score of last 50 episodes
          #every 50th episode print score
          if episode%50 == 0:
              print('Episode {}\tScore: {}\tAverage score(last 50 episodes): {:.2f}'.format(episode,scores[-50:],mean_score))
  
      
      #print('Losses',Losses)
      Losses = torch.stack(Losses).numpy()
      #print('Losses',Losses)
      plt.plot(np.arange(len(Losses)),Losses)
      plt.xlabel('Training iterations')
      plt.ylabel('Loss')
      plt.show()
      
      Bellman = np.array(Bellman)
      #print('Bellman',Bellman,'\n')
      plt.plot(np.arange(len(Bellman)),Bellman)
      plt.xlabel('Training iterations')
      plt.ylabel('Bellman target')
      plt.show()
      
      #print('scores',scores)
      plt.plot(np.arange(len(scores)),scores)
      plt.xlabel('Training iterations')
      plt.ylabel('Reward')
      plt.show()
      
      #print('epsilon',Epsilon)
      plt.plot(np.arange(len(Epsilon)),Epsilon)
      plt.xlabel('Training iterations')
      plt.ylabel('Epsilon')
      plt.show()
      
      print('Times',Times[-25:])
      print('Deltas',Deltas[-25:])
      
      Times = np.array(Times)
      print('Times',Times)
      #plt.figure(figsize=(31,20))
      plt.figure(figsize=(9,7))
      plt.plot(np.arange(len(Times)),(np.array(Times)))
      plt.xlabel('Training iterations')
      plt.ylabel('t')
      plt.show()
       
      Times_1 = np.array(Times[-300:])
      print('t',Times)
      plt.figure(figsize=(9,7))
      plt.plot(np.arange(len(Times_1)),(np.array(Times_1)))
      plt.xlabel('Last 300 Training iterations')
      plt.ylabel('t')
      plt.ylim(0,1000)
      plt.show()
      
  model = DqnNet()#policy         
  memory = ReplayMemory(20000)
  optimizer = optim.Adam(model.parameters(), lr=learning_rate)
  
  train(10000)
  
  ```
  
  **Tags:** reinforcement-learning, q-learning, dqn
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-10-11. Score: 3
  
  I see some issues in your code of the environment. Firstly, and probably most importantly, you should not be incrementing the reward. In your code, every time the agent hits $t=475$ for example, the reward given by the environment increases by 1. So if the agent oscillates between $t=450$ and $t=475$, at each timestep the environment gives a greater and greater reward. Your Q values have no reason to converge. Instead of incrementing the reward, you should just return a reward of $1$ at those two desired states and return $0$ otherwise.
  
  Secondly, it appears that you don't reset the state during `reset()`, I think you should have `self.time = 0` in there (unless this was intentional).
  
  **Edit** : I also question why you're using DQN for this problem. This problem should be very easy to solve with standard (tabular) Q Learning.
  
  **Edit 2** : Your training loop looks wrong. You reset the environment at each iteration in your loop, so your episodes are one timestep long each. What I think you want instead is to have another loop under `reset()` something like
  
  ```
  for episode in range(episodes):
    s = env.reset()
    episode_score = 0 
    for step in range(max_timesteps):
      a = agent.predict_action(s)
      ns, r, d, _ = env.step(a)
      episode_score += r
     ... 
      agent.update_policy()
     ... 
    scores.append(episode_score)
   ... 
  
  ```
  
  > **Comments:**
  > **username_0**: thanks, I just corrected the typo in the reset function and changed the reward scheme.. unfortunately, still no convergence towards t = 475 or 450 ... and to answer your question: i am still new to reinforcement learning, and wanted to learn how to implement DQN  on a simple problem, just for the sake of learning and understanding  
  > **username_1**: I also suggest you increase gamma. Gamma=0.5 is fairly low, and it's very possible that the rewards are just not propagating back to other states due to the exponential decay. Try with gamma=0.99 or something. Not sure if this will be the definitive solution, but it can't hurt.  
  > **username_1**: Also, can you describe the results you get? As in, what do you mean by "no convergence towards t=..."? Can you talk about what you expect vs what you observe?  
  > **username_0**: thank you, so let me explain: in the end i want the agent to do the following 475 +0 = 475 --> 475 + 0 = 475 --> 475 +0 = 475.... OR something like 475 -25 = 450 --> 450 +25 = 475 --> 475 -25 = 450 .... (i want the sum to always be 475 or 450)... what I observe: it just takes the same action for any state ( for example 450 +25 = 475 --> 475 + 25 = 500 -->  525 + 25 = 550 --> 550 + 25 = 575....).. so it just learns to take always the same action over and over again without caring about the result  
  > **username_1**: Noticed another issue, updated my original answer.  
  > **username_0**: Thank you very much for your comments and suggestions.. unfortunately i still don't get what I expect the agent to learn ( i changed the hyperparameters too, but no success so far)... probably DQN is not very well suited for this type of task as you mentioned  
  > **username_1**: If you're just doing this as a learning exercise, why don't you try an openai gym environment? DQN definitely can solve cartpole, so I suggest you try that to figure out if there's something wrong with your DQN implementation.  
  
  ---
  
  