Body: <p><strong>TL;DR: read the bold. The rest are details</strong></p>
  
  <p><strong>I am trying to implement</strong> <a href="http://incompleteideas.net/book/bookdraft2017nov5.pdf" rel="nofollow noreferrer">Reinforcement Learning:An Introduction, section 13.5</a> <strong>myself</strong>:</p>
  
  <p><a href="https://i.sstatic.net/JDK5k.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/JDK5k.png" alt="enter image description here"></a></p>
  
  <p>on <a href="https://gym.openai.com/envs/CartPole-v0/" rel="nofollow noreferrer">OpenAi's cartpole</a></p>
  
  <p><strong>The algorithm seems to be learning something useful</strong> (and not random), as shown in these graphs (different zoom on the same run):</p>
  
  <p><a href="https://i.sstatic.net/wkyXW.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/wkyXW.png" alt="enter image description here"></a></p>
  
  <p><a href="https://i.sstatic.net/ZWAxU.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ZWAxU.png" alt="enter image description here"></a></p>
  
  <p><a href="https://i.sstatic.net/Sxkmf.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Sxkmf.png" alt="enter image description here"></a></p>
  
  <p>Which show the reward per episode (y axis is the "time alive", x axis is episode number).</p>
  
  <p><strong>However</strong>, as can be seen, </p>
  
  <ol>
  <li><p>The learning does not seem to stabilize.</p></li>
  <li><p>It looks like every time the reward maxes out (200), it immediately drops.</p></li>
  </ol>
  
  <hr>
  
  <p><strong>My relevant code for reference</strong> (inspired by <a href="https://github.com/pytorch/examples/blob/master/reinforcement_learning/actor_critic.py" rel="nofollow noreferrer">pytorch's actor critic</a>)</p>
  
  <p>note: <em>in this question, <code>xp_batch</code> is ONLY THE VERY LAST (s, a, r, s'), meaning experience replay is not in use in this code</em>!</p>
  
  <p>The actor and critic are both distinct neural networks which </p>
  
  <pre><code>def learn(self, xp_batch):#in this question, xp_batch is ONLY THE VERY LAST (s, a, r, s')
      for s_t, a_t, r_t, s_t1 in xp_batch:
          expected_reward_from_t = self.critic_nn(s_t)
          probs_t = self.actor_nn(s_t)
          expected_reward_from_t1 = torch.tensor([[0]], dtype=torch.float)
          if s_t1 is not None:  # s_t is not a terminal state, s_t1 exists.
              expected_reward_from_t1 = self.critic_nn(s_t1)
  
          m = Categorical(probs_t)
          log_prob_t = m.log_prob(a_t)
  
          delta = r_t + self.args.gamma * expected_reward_from_t1 - expected_reward_from_t
  
          loss_critic = delta * expected_reward_from_t
          self.critic_optimizer.zero_grad()
          loss_critic.backward(retain_graph=True)
          self.critic_optimizer.step()
  
          delta.detach()
          loss_actor = delta * log_prob_t
          self.actor_optimizer.zero_grad()
          loss_actor.backward()
          self.actor_optimizer.step()
  
  def select_action(self, state):
      probs = self.actor_nn(state)
      m = Categorical(probs)
      action  = m.sample()
      return action
  </code></pre>
  
  <hr>
  
  <p><strong>My questions are:</strong></p>
  
  <ol>
  <li><p><strong>Am I doing something wrong, or is this to be expected</strong>?</p></li>
  <li><p>I know this can be improved with eligibility traces/experience replay+off policy learning. <strong>Before making those upgrades, I want to make sure the current results make sense.</strong></p></li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-14T22:26:24.660Z
Comments.Text: I think that questions related to programming issues are better suited for https://datascience.stackexchange.com/ or Stack Overflow. Anyway, I've you tried to run the PyTorch example (which you based your implementation on) and see if it behaves similarly to yours?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-14T22:40:13.437Z
Comments.Text: The example I linked to uses full episodes discounted returns. Mine is (supposed to be) using immediate rewards, online, regardless of episodes. just the very last sample.
  So, they aren't supposed to give similar results, which is why I come to the experts and ask if this makes any sense
Comments.UserDisplayName: Gulzar
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-14T22:44:40.043Z
Comments.Text: @nbro If this is better in datascience.stackexchange.com, I don't mind it being moved, should I ask for it via flagging? Anyway, this question is more about the results (in the graphs) than the code. The code is there for reference.
Comments.UserDisplayName: Gulzar
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-14T22:46:07.720Z
Comments.Text: You could simply copy the text of this question to another question on Data Science SE and then (if you want) delete the question from here. Hopefully, there you will find more help, but I can't guarantee that.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-14T22:57:55.470Z
Comments.Text: Anyway, I feel like you're attempting to implement some non-trivial algorithms and trying to combine different techniques without really knowing the details of these algorithms and techniques or fully understanding them. This will make your life harder. I would encourage you to first implement what is really known to work. Then try to formalise what you think might work. Only then you should attempt to implement it.
Comments.UserDisplayName: nbro
Tags: <machine-learning><reinforcement-learning><pytorch>
Title: How to make episode ending "good" in reinforcement learning?
UserDisplayName: Gulzar
ThreadText: # How to make episode ending "good" in reinforcement learning?
  
  (Asked by: username_0 on 2019-02-14)
  
  TL;DR: read the bold. The rest are details
  I am trying to implement Reinforcement Learning:An Introduction, section 13.5 myself:
  
  on OpenAi's cartpole
  The algorithm seems to be learning something useful (and not random), as shown in these graphs (different zoom on the same run):
  
  
  
  Which show the reward per episode (y axis is the "time alive", x axis is episode number).
  However, as can be seen, 
  
  The learning does not seem to stabilize.
  It looks like every time the reward maxes out (200), it immediately drops.
  
  
  My relevant code for reference (inspired by pytorch's actor critic)
  note: in this question, xp_batch is ONLY THE VERY LAST (s, a, r, s'), meaning experience replay is not in use in this code!
  The actor and critic are both distinct neural networks which 
  def learn(self, xp_batch):#in this question, xp_batch is ONLY THE VERY LAST (s, a, r, s')
      for s_t, a_t, r_t, s_t1 in xp_batch:
          expected_reward_from_t = self.critic_nn(s_t)
          probs_t = self.actor_nn(s_t)
          expected_reward_from_t1 = torch.tensor([[0]], dtype=torch.float)
          if s_t1 is not None:  # s_t is not a terminal state, s_t1 exists.
              expected_reward_from_t1 = self.critic_nn(s_t1)
  
          m = Categorical(probs_t)
          log_prob_t = m.log_prob(a_t)
  
          delta = r_t + self.args.gamma * expected_reward_from_t1 - expected_reward_from_t
  
          loss_critic = delta * expected_reward_from_t
          self.critic_optimizer.zero_grad()
          loss_critic.backward(retain_graph=True)
          self.critic_optimizer.step()
  
          delta.detach()
          loss_actor = delta * log_prob_t
          self.actor_optimizer.zero_grad()
          loss_actor.backward()
          self.actor_optimizer.step()
  
  def select_action(self, state):
      probs = self.actor_nn(state)
      m = Categorical(probs)
      action  = m.sample()
      return action
  
  
  My questions are:
  
  Am I doing something wrong, or is this to be expected?
  I know this can be improved with eligibility traces/experience replay+off policy learning. Before making those upgrades, I want to make sure the current results make sense.
  
  **Tags:** machine-learning, reinforcement-learning, pytorch
  > **Comments:**
  > **username_1**: I think that questions related to programming issues are better suited for https://datascience.stackexchange.com/ or Stack Overflow. Anyway, I've you tried to run the PyTorch example (which you based your implementation on) and see if it behaves similarly to yours?  
  > **username_0**: The example I linked to uses full episodes discounted returns. Mine is (supposed to be) using immediate rewards, online, regardless of episodes. just the very last sample. So, they aren't supposed to give similar results, which is why I come to the experts and ask if this makes any sense  
  > **username_0**: @nbro If this is better in datascience.stackexchange.com, I don't mind it being moved, should I ask for it via flagging? Anyway, this question is more about the results (in the graphs) than the code. The code is there for reference.  
  > **username_1**: You could simply copy the text of this question to another question on Data Science SE and then (if you want) delete the question from here. Hopefully, there you will find more help, but I can't guarantee that.  
  > **username_1**: Anyway, I feel like you're attempting to implement some non-trivial algorithms and trying to combine different techniques without really knowing the details of these algorithms and techniques or fully understanding them. This will make your life harder. I would encourage you to first implement what is really known to work. Then try to formalise what you think might work. Only then you should attempt to implement it.  
  
  ---
  
  ## 0 answers
  
  