Body: <p>In the original DQN paper, gradients during training are derived as follows:</p>
  <p><span class="math-container">$\nabla_{\theta_i} L_i\left(\theta_i\right)=\mathbb{E}_{s, a \sim \rho(\cdot) ; s^{\prime} \sim \mathcal{E}}\left[\left(r+\gamma \max _{a^{\prime}} Q\left(s^{\prime}, a^{\prime} ; \theta_{i-1}\right)-Q\left(s, a ; \theta_i\right)\right) \nabla_{\theta_i} Q\left(s, a ; \theta_i\right)\right]$</span>.</p>
  <p>However, I find that replacing the max operator with a random selection works just as well. In what circumstances might this hold?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-21T08:23:09.043Z
Comments.Text: So, basically, you're using a random policy to select an action from $s'$? It might be a good idea to provide more context about your specific problem and what you mean by "works just as well".
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-22T23:12:27.193Z
Comments.Text: Instead of computing all $Q(s^{\prime}, a^{\prime})$ and taking the largest of those values, I find that computing the value for a single randomly selected $a^{\prime}$ works just as well.
Comments.UserDisplayName: bonzo_pippinpaddle
Tags: <reinforcement-learning><dqn>
Title: In what circumstances can we replace the max operator with random selection in the DQN?
UserDisplayName: bonzo_pippinpaddle
ThreadText: # In what circumstances can we replace the max operator with random selection in the DQN?
  
  (Asked by: username_0 on 2022-12-19)
  
  In the original DQN paper, gradients during training are derived as follows:
  
  $\nabla\_{\theta\_i} L\_i\left(\theta\_i\right)=\mathbb{E}\_{s, a \sim \rho(\cdot) ; s^{\prime} \sim \mathcal{E}}\left[\left(r+\gamma \max \_{a^{\prime}} Q\left(s^{\prime}, a^{\prime} ; \theta\_{i-1}\right)-Q\left(s, a ; \theta\_i\right)\right) \nabla\_{\theta\_i} Q\left(s, a ; \theta\_i\right)\right]$.
  
  However, I find that replacing the max operator with a random selection works just as well. In what circumstances might this hold?
  
  **Tags:** reinforcement-learning, dqn
  
  > **Comments:**
  > **username_1**: So, basically, you're using a random policy to select an action from $s'$? It might be a good idea to provide more context about your specific problem and what you mean by "works just as well".  
  > **username_0**: Instead of computing all $Q(s^{\prime}, a^{\prime})$ and taking the largest of those values, I find that computing the value for a single randomly selected $a^{\prime}$ works just as well.  
  
  ---
  
  ## 0 answers
  
  