Answers.Body: <p>Cartpole will work just fine with a single-step TD method - e.g. Q learning - and a simple neural network. <em>Configured well</em>, a simple DQN model should solve it in a few minutes. You can also forgo the neural network and use discretised states, or tile coding etc.</p>
  
  <p>What you cannot do is plug a neural network estimator into basic Q learning and make no other adjustments. That is because the bootstrap process in TD learning will create a runaway feedback driven from the random and incorrect return estimates in the initialised NN. You have to use something like DQN to get it to work at all. This is different to Monte Carlo approach, where you can generally just plug in a NN in place of a Q table.</p>
  
  <p>The basics of DQN are:</p>
  
  <ul>
  <li><p>Use experience replay. Store <span class="math-container">$(S, A, R, S')$</span> values from each step, and when training the NN, draw a small random batch (e.g. maybe 16 samples) from this table, re-estimate their Q values to calculate TD target e.g.  <span class="math-container">$R + \gamma\text{max}_{a'} Q(S',a')$</span> and use that to create a minibatch to train the NN </p></li>
  <li><p>Use a "delayed" target network for estimating TD target. This can just be a snapshot of your learning network, taken once every N steps, where N is typically anything from 100 to 100000.</p></li>
  </ul>
  
  <p>Further details of what <em>configured well</em> means varies from problem to problem. With DQN you will want to play with hyper parameters of experience history size, time to keep frozen copy of network (for bootstrap estimates), batch size of experience table samples, amount of exploration. </p>
  
  <p>In CartPole though, there will be a wide range of acceptable hyperparameter values. So I suspect you have just plugged in a NN in place of the Q table and wonder why it doesn't work.</p>
  
  <blockquote>
    <p>I assume that is the case because in Cartpole, for every timestamp, we get a reward of 1, which has very little immediate information about weather or not the action was good or not.</p>
  </blockquote>
  
  <p>Yes this is part of what makes some learning goals become harder challenges in sequential-decision control systems. However, this was also the case for your Monte Carlo method, and in general this relates to the <a href="http://www.bcp.psych.ualberta.ca/~mike/Pearl_Street/Dictionary/contents/C/creditassign.html" rel="nofollow noreferrer">credit assignment problem</a> in RL.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-02T17:26:03.087Z
Answers.Text: Thanks :) I will go on to implementing what you suggested
Answers.UserDisplayName: Gulzar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-02T18:31:29.950Z
Answers.Text: Also, why does having the "delayed" target network help stability? after all, we take an action, and receive a reward. Why does it matter if that action and reward came from a (temporarily) constant distribution or not?
Answers.UserDisplayName: Gulzar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-02T18:44:04.717Z
Answers.Text: Also by the way, When drawing an experience whose `s` is terminal (meaning no s'), should I just plug in 0 (zero) in place of Q(s',a')?
Answers.UserDisplayName: Gulzar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-02T19:47:53.880Z
Answers.Text: @Guizar: Yes, plug in a zero, the value of $Q(s_T, \cdot)$ is 0 by definition when $s_T$ is a terminal state, because Q is predicting future rewards and there are none. The delayed network is a separate question, too much detail for a comment - I think I have answered a similar one here on the site, sorry cannot find it at the moment
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-02T17:12:40.057Z
Answers.LastActivityDate: 2019-02-02T17:12:40.057Z
Answers.UserDisplayName: Neil Slater
Body: <p>Up to now, I have been using (my version of) <a href="https://github.com/pytorch/examples/blob/master/reinforcement_learning/actor_critic.py" rel="nofollow noreferrer">open AI's code</a>, with the suggested CartPole.</p>
  
  <p>I have been using Monte Carlo methods, which, for cartpole, seemed to work fine.</p>
  
  <p>Trying to move to temporal difference, Cartpole seems to fail to learn (with simple TD method) (or I stopped it too soon, but still unacceptable).</p>
  
  <p>I assume that is the case because in Cartpole, for every timestamp, we get a reward of 1, which has very little immediate information about weather or not the action was good or not.</p>
  
  <p><strong>Which gym environment is the simplest that would probably work with TD learning?</strong></p>
  
  <p>by simplest I mean that there is no need for a large NN to solve it. No conv nets, no RNNS. just a few small layers of a fully connected NN, just like in cartpole. something I can train on my home cpu, just to see it starting to converge.</p>
  
Tags: <machine-learning><reinforcement-learning><open-ai>
Title: Which Openai Gym environment should I use to test a Temporal Difference RL Agent?
UserDisplayName: Gulzar
ThreadText: # Which Openai Gym environment should I use to test a Temporal Difference RL Agent?
  
  (Asked by: username_0 on 2019-02-02)
  
  Up to now, I have been using (my version of) [open AI's code](https://github.com/pytorch/examples/blob/master/reinforcement_learning/actor_critic.py), with the suggested CartPole.
  
  I have been using Monte Carlo methods, which, for cartpole, seemed to work fine.
  
  Trying to move to temporal difference, Cartpole seems to fail to learn (with simple TD method) (or I stopped it too soon, but still unacceptable).
  
  I assume that is the case because in Cartpole, for every timestamp, we get a reward of 1, which has very little immediate information about weather or not the action was good or not.
  
  **Which gym environment is the simplest that would probably work with TD learning?**
  
  by simplest I mean that there is no need for a large NN to solve it. No conv nets, no RNNS. just a few small layers of a fully connected NN, just like in cartpole. something I can train on my home cpu, just to see it starting to converge.
  
  **Tags:** machine-learning, reinforcement-learning, open-ai
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-02-02. Score: 1
  
  Cartpole will work just fine with a single-step TD method - e.g. Q learning - and a simple neural network. *Configured well*, a simple DQN model should solve it in a few minutes. You can also forgo the neural network and use discretised states, or tile coding etc.
  
  What you cannot do is plug a neural network estimator into basic Q learning and make no other adjustments. That is because the bootstrap process in TD learning will create a runaway feedback driven from the random and incorrect return estimates in the initialised NN. You have to use something like DQN to get it to work at all. This is different to Monte Carlo approach, where you can generally just plug in a NN in place of a Q table.
  
  The basics of DQN are:
  
  * Use experience replay. Store $(S, A, R, S')$ values from each step, and when training the NN, draw a small random batch (e.g. maybe 16 samples) from this table, re-estimate their Q values to calculate TD target e.g. $R + \gamma\text{max}\_{a'} Q(S',a')$ and use that to create a minibatch to train the NN
  * Use a "delayed" target network for estimating TD target. This can just be a snapshot of your learning network, taken once every N steps, where N is typically anything from 100 to 100000.
  
  Further details of what *configured well* means varies from problem to problem. With DQN you will want to play with hyper parameters of experience history size, time to keep frozen copy of network (for bootstrap estimates), batch size of experience table samples, amount of exploration.
  
  In CartPole though, there will be a wide range of acceptable hyperparameter values. So I suspect you have just plugged in a NN in place of the Q table and wonder why it doesn't work.
  
  > I assume that is the case because in Cartpole, for every timestamp, we get a reward of 1, which has very little immediate information about weather or not the action was good or not.
  
  Yes this is part of what makes some learning goals become harder challenges in sequential-decision control systems. However, this was also the case for your Monte Carlo method, and in general this relates to the [credit assignment problem](http://www.bcp.psych.ualberta.ca/~mike/Pearl_Street/Dictionary/contents/C/creditassign.html) in RL.
  
  > **Comments:**
  > **username_0**: Thanks :) I will go on to implementing what you suggested  
  > **username_0**: Also, why does having the "delayed" target network help stability? after all, we take an action, and receive a reward. Why does it matter if that action and reward came from a (temporarily) constant distribution or not?  
  > **username_0**: Also by the way, When drawing an experience whose `s` is terminal (meaning no s'), should I just plug in 0 (zero) in place of Q(s',a')?  
  > **username_1**: @Guizar: Yes, plug in a zero, the value of $Q(s_T, \cdot)$ is 0 by definition when $s_T$ is a terminal state, because Q is predicting future rewards and there are none. The delayed network is a separate question, too much detail for a comment - I think I have answered a similar one here on the site, sorry cannot find it at the moment  
  
  ---
  
  