<p>Just so that this could be useful for people who refer to this post later on: Please refer to Sutton's reinforcement learning book (2nd edition) example 11.2. It provides an example for why full gradient wouldn't work.</p>

CC BY-SA 4.0
2021-05-20T22:54:14.003Z
2021-05-20T22:54:14.003Z
Vikranth
<blockquote>
  <p>Semi gradient methods work well in <strong>Reinforcement Learning</strong>, but what is the reason of <strong><em>not using</em></strong> the true gradient if it can be computed? </p>
</blockquote>

<p>Just complexity and extra computation, in many cases for a marginal benefit.</p>

<blockquote>
  <p>I tried it on the cart pole problem with a deep Q-Network and it performed much worse than traditional semi gradient, is there a concrete reason for this?</p>
</blockquote>

<p>It is hard to tell, without exploring the implementation in detail. However, DQN is an inherently unstable learning technique that needs care in choosing hyper-parameters that control this instability and offset against learning rate:</p>

<ul>
<li>size of minibatch to train from experience replay on each step</li>
<li>number of training steps between taking frozen copies for estimation*</li>
<li>whether or not you use double-learning to avoid maximisation bias (more important if you have fine-grained discretisation of continuous action space)</li>
</ul>

<p>There is a chance that the optimal choices here are different between true gradient and semi gradient approaches.</p>

<p>* The frozen estimator could be a big clue here in your implementation. If you are using this frozen copy technique, it has a big impact on how you should calculate the true gradient, because changing the parameters would no longer change the current TD target - which is what the true gradient approach fixes. However, getting rid of this stability-improving addition in order to get true gradients might on balance make the algorithm less stable - you could try to fix that by taking larger mini-batches.</p>

CC BY-SA 3.0
2018-04-26T07:25:03.200Z
2018-04-26T07:25:03.200Z
Neil Slater
<p>Semi-gradient methods work well in reinforcement learning, but what is there a reason of <strong>not</strong> using the true gradient if it can be computed? </p>

<p>I tried it on the cart pole problem with a deep Q-network and it performed much worse than traditional semi-gradient. Is there a concrete reason for this?</p>

<reinforcement-learning><gradient-descent><dqn><deep-rl><function-approximation>
Why use semi-gradient instead of full gradient in RL problems, when using function approximation?
Nestor Sanchez
# Why use semi-gradient instead of full gradient in RL problems, when using function approximation?

(Asked by: username_0 on 2018-04-24)

Semi-gradient methods work well in reinforcement learning, but what is there a reason of **not** using the true gradient if it can be computed?

I tried it on the cart pole problem with a deep Q-network and it performed much worse than traditional semi-gradient. Is there a concrete reason for this?

**Tags:** reinforcement-learning, gradient-descent, dqn, deep-rl, function-approximation


---

## 2 answers

### Answer by username_1 on 2021-05-20. Score: 2

Just so that this could be useful for people who refer to this post later on: Please refer to Sutton's reinforcement learning book (2nd edition) example 11.2. It provides an example for why full gradient wouldn't work.


---

### Answer by username_2 on 2018-04-26. Score: 1

> Semi gradient methods work well in **Reinforcement Learning**, but what is the reason of ***not using*** the true gradient if it can be computed?

Just complexity and extra computation, in many cases for a marginal benefit.

> I tried it on the cart pole problem with a deep Q-Network and it performed much worse than traditional semi gradient, is there a concrete reason for this?

It is hard to tell, without exploring the implementation in detail. However, DQN is an inherently unstable learning technique that needs care in choosing hyper-parameters that control this instability and offset against learning rate:

* size of minibatch to train from experience replay on each step
* number of training steps between taking frozen copies for estimation\*
* whether or not you use double-learning to avoid maximisation bias (more important if you have fine-grained discretisation of continuous action space)

There is a chance that the optimal choices here are different between true gradient and semi gradient approaches.

\* The frozen estimator could be a big clue here in your implementation. If you are using this frozen copy technique, it has a big impact on how you should calculate the true gradient, because changing the parameters would no longer change the current TD target - which is what the true gradient approach fixes. However, getting rid of this stability-improving addition in order to get true gradients might on balance make the algorithm less stable - you could try to fix that by taking larger mini-batches.


---

