<p>In the original DQN paper, gradients during training are derived as follows:</p>
<p><span class="math-container">$\nabla_{\theta_i} L_i\left(\theta_i\right)=\mathbb{E}_{s, a \sim \rho(\cdot) ; s^{\prime} \sim \mathcal{E}}\left[\left(r+\gamma \max _{a^{\prime}} Q\left(s^{\prime}, a^{\prime} ; \theta_{i-1}\right)-Q\left(s, a ; \theta_i\right)\right) \nabla_{\theta_i} Q\left(s, a ; \theta_i\right)\right]$</span>.</p>
<p>However, I find that replacing the max operator with a random selection works just as well. In what circumstances might this hold?</p>

CC BY-SA 4.0
2022-12-21T08:23:09.043Z
So, basically, you're using a random policy to select an action from $s'$? It might be a good idea to provide more context about your specific problem and what you mean by "works just as well".
nbro
CC BY-SA 4.0
2022-12-22T23:12:27.193Z
Instead of computing all $Q(s^{\prime}, a^{\prime})$ and taking the largest of those values, I find that computing the value for a single randomly selected $a^{\prime}$ works just as well.
bonzo_pippinpaddle
<reinforcement-learning><dqn>
In what circumstances can we replace the max operator with random selection in the DQN?
bonzo_pippinpaddle
# In what circumstances can we replace the max operator with random selection in the DQN?

(Asked by: username_0 on 2022-12-19)

In the original DQN paper, gradients during training are derived as follows:

$\nabla\_{\theta\_i} L\_i\left(\theta\_i\right)=\mathbb{E}\_{s, a \sim \rho(\cdot) ; s^{\prime} \sim \mathcal{E}}\left[\left(r+\gamma \max \_{a^{\prime}} Q\left(s^{\prime}, a^{\prime} ; \theta\_{i-1}\right)-Q\left(s, a ; \theta\_i\right)\right) \nabla\_{\theta\_i} Q\left(s, a ; \theta\_i\right)\right]$.

However, I find that replacing the max operator with a random selection works just as well. In what circumstances might this hold?

**Tags:** reinforcement-learning, dqn

> **Comments:**
> **username_1**: So, basically, you're using a random policy to select an action from $s'$? It might be a good idea to provide more context about your specific problem and what you mean by "works just as well".  
> **username_0**: Instead of computing all $Q(s^{\prime}, a^{\prime})$ and taking the largest of those values, I find that computing the value for a single randomly selected $a^{\prime}$ works just as well.  

---

## 0 answers

