Answers.Body: <p>Just so that this could be useful for people who refer to this post later on: Please refer to Sutton's reinforcement learning book (2nd edition) example 11.2. It provides an example for why full gradient wouldn't work.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-20T22:54:14.003Z
Answers.LastActivityDate: 2021-05-20T22:54:14.003Z
Answers.UserDisplayName: Vikranth
Answers.Body: <blockquote>
    <p>Semi gradient methods work well in <strong>Reinforcement Learning</strong>, but what is the reason of <strong><em>not using</em></strong> the true gradient if it can be computed? </p>
  </blockquote>
  
  <p>Just complexity and extra computation, in many cases for a marginal benefit.</p>
  
  <blockquote>
    <p>I tried it on the cart pole problem with a deep Q-Network and it performed much worse than traditional semi gradient, is there a concrete reason for this?</p>
  </blockquote>
  
  <p>It is hard to tell, without exploring the implementation in detail. However, DQN is an inherently unstable learning technique that needs care in choosing hyper-parameters that control this instability and offset against learning rate:</p>
  
  <ul>
  <li>size of minibatch to train from experience replay on each step</li>
  <li>number of training steps between taking frozen copies for estimation*</li>
  <li>whether or not you use double-learning to avoid maximisation bias (more important if you have fine-grained discretisation of continuous action space)</li>
  </ul>
  
  <p>There is a chance that the optimal choices here are different between true gradient and semi gradient approaches.</p>
  
  <p>* The frozen estimator could be a big clue here in your implementation. If you are using this frozen copy technique, it has a big impact on how you should calculate the true gradient, because changing the parameters would no longer change the current TD target - which is what the true gradient approach fixes. However, getting rid of this stability-improving addition in order to get true gradients might on balance make the algorithm less stable - you could try to fix that by taking larger mini-batches.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-26T07:25:03.200Z
Answers.LastActivityDate: 2018-04-26T07:25:03.200Z
Answers.UserDisplayName: Neil Slater
Body: <p>Semi-gradient methods work well in reinforcement learning, but what is there a reason of <strong>not</strong> using the true gradient if it can be computed? </p>
  
  <p>I tried it on the cart pole problem with a deep Q-network and it performed much worse than traditional semi-gradient. Is there a concrete reason for this?</p>
  
Tags: <reinforcement-learning><gradient-descent><dqn><deep-rl><function-approximation>
Title: Why use semi-gradient instead of full gradient in RL problems, when using function approximation?
UserDisplayName: Nestor Sanchez
ThreadText: # Why use semi-gradient instead of full gradient in RL problems, when using function approximation?
  
  (Asked by: username_0 on 2018-04-24)
  
  Semi-gradient methods work well in reinforcement learning, but what is there a reason of **not** using the true gradient if it can be computed?
  
  I tried it on the cart pole problem with a deep Q-network and it performed much worse than traditional semi-gradient. Is there a concrete reason for this?
  
  **Tags:** reinforcement-learning, gradient-descent, dqn, deep-rl, function-approximation
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-05-20. Score: 2
  
  Just so that this could be useful for people who refer to this post later on: Please refer to Sutton's reinforcement learning book (2nd edition) example 11.2. It provides an example for why full gradient wouldn't work.
  
  
  ---
  
  ### Answer by username_2 on 2018-04-26. Score: 1
  
  > Semi gradient methods work well in **Reinforcement Learning**, but what is the reason of ***not using*** the true gradient if it can be computed?
  
  Just complexity and extra computation, in many cases for a marginal benefit.
  
  > I tried it on the cart pole problem with a deep Q-Network and it performed much worse than traditional semi gradient, is there a concrete reason for this?
  
  It is hard to tell, without exploring the implementation in detail. However, DQN is an inherently unstable learning technique that needs care in choosing hyper-parameters that control this instability and offset against learning rate:
  
  * size of minibatch to train from experience replay on each step
  * number of training steps between taking frozen copies for estimation\*
  * whether or not you use double-learning to avoid maximisation bias (more important if you have fine-grained discretisation of continuous action space)
  
  There is a chance that the optimal choices here are different between true gradient and semi gradient approaches.
  
  \* The frozen estimator could be a big clue here in your implementation. If you are using this frozen copy technique, it has a big impact on how you should calculate the true gradient, because changing the parameters would no longer change the current TD target - which is what the true gradient approach fixes. However, getting rid of this stability-improving addition in order to get true gradients might on balance make the algorithm less stable - you could try to fix that by taking larger mini-batches.
  
  
  ---
  
  