Answers.Body: <p>The loss function for DQN algorithm is
  <span class="math-container">\begin{equation}
  L(\theta_i) = \mathbb E_{s, a, r, s'} [(y - Q(s, a;\theta_i))^2]
  \end{equation}</span>
  Like you said, we only take one action per timestep. We can only shift weights of the network that had the effect in calculating action value <span class="math-container">$Q(s, a)$</span> for that particular action that we took. For that action, variable <span class="math-container">$y$</span> would have value
  <span class="math-container">\begin{equation}
  y = r + \gamma \max_{a'} Q(s', a', \theta^-_i)
  \end{equation}</span>
  and we would have standard form of DQN loss. We calculate the gradient of that loss with respect to network parameters <span class="math-container">$\theta$</span>, backprop it and slightly shift weights so to more accurately estimate <span class="math-container">$Q(s, a)$</span> for that specific state-action pair.</p>
  
  <p>We never took other actions. Since we never took them we didn't get reward <span class="math-container">$r$</span> and we can't estimate <span class="math-container">$\max_{a'} Q(s', a', \theta^-_i)$</span> for those other actions. We don't want to change weights that had effect on calculating actions values for those other actions because we have no way of estimating how accurate they were. Standard approach is to set
  <span class="math-container">\begin{equation}
  y = Q(s, a;\theta_i)
  \end{equation}</span>
  this way the loss for those other actions would be <span class="math-container">$0$</span> and it would result in not changing the weights that had influence in calculating their action values.  </p>
  
  <p>Setting target <span class="math-container">$y$</span> to <span class="math-container">$0$</span> for all other actions would mean that we want <span class="math-container">$Q(s, a)$</span> for them all to slightly shift to <span class="math-container">$0$</span>. That would not be correct since we have no way of knowing their true value. I think you misinterpreted that part in the implementations.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-18T16:06:31.723Z
Answers.LastActivityDate: 2019-09-18T16:06:31.723Z
Answers.UserDisplayName: Brale
Body: <p>Let's say that we have three actions. The highest-valued action of the three choices is the first. When training the DQN, what do we do with the other two, as we don't have a target for them, since they weren't taken?</p>
  <p>I've seen some code that leaves the target for off actions as whatever the prediction returned, which feels a bit wrong to me as two or more similar behaving actions might never be differentiated well after random action selection dwindles.</p>
  <p>I've also seen some implementations that set the target for all actions to zero and only adjust the target for the action taken. This would help regarding action differentiation long term, but it also puts more reliance on taking random actions for any unfamiliar states (I believe) as an off action might never be taken otherwise.</p>
  
Tags: <reinforcement-learning><training><q-learning><dqn><deep-rl>
Title: When training a DQN, how should we update the value of actions that were not taken?
UserDisplayName: Mythics
ThreadText: # When training a DQN, how should we update the value of actions that were not taken?
  
  (Asked by: username_0 on 2019-09-18)
  
  Let's say that we have three actions. The highest-valued action of the three choices is the first. When training the DQN, what do we do with the other two, as we don't have a target for them, since they weren't taken?
  
  I've seen some code that leaves the target for off actions as whatever the prediction returned, which feels a bit wrong to me as two or more similar behaving actions might never be differentiated well after random action selection dwindles.
  
  I've also seen some implementations that set the target for all actions to zero and only adjust the target for the action taken. This would help regarding action differentiation long term, but it also puts more reliance on taking random actions for any unfamiliar states (I believe) as an off action might never be taken otherwise.
  
  **Tags:** reinforcement-learning, training, q-learning, dqn, deep-rl
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-09-18. Score: 3 | Accepted answer
  
  The loss function for DQN algorithm is
  \begin{equation}
  L(\theta\_i) = \mathbb E\_{s, a, r, s'} [(y - Q(s, a;\theta\_i))^2]
  \end{equation}
  Like you said, we only take one action per timestep. We can only shift weights of the network that had the effect in calculating action value $Q(s, a)$ for that particular action that we took. For that action, variable $y$ would have value
  \begin{equation}
  y = r + \gamma \max\_{a'} Q(s', a', \theta^-\_i)
  \end{equation}
  and we would have standard form of DQN loss. We calculate the gradient of that loss with respect to network parameters $\theta$, backprop it and slightly shift weights so to more accurately estimate $Q(s, a)$ for that specific state-action pair.
  
  We never took other actions. Since we never took them we didn't get reward $r$ and we can't estimate $\max\_{a'} Q(s', a', \theta^-\_i)$ for those other actions. We don't want to change weights that had effect on calculating actions values for those other actions because we have no way of estimating how accurate they were. Standard approach is to set
  \begin{equation}
  y = Q(s, a;\theta\_i)
  \end{equation}
  this way the loss for those other actions would be $0$ and it would result in not changing the weights that had influence in calculating their action values.
  
  Setting target $y$ to $0$ for all other actions would mean that we want $Q(s, a)$ for them all to slightly shift to $0$. That would not be correct since we have no way of knowing their true value. I think you misinterpreted that part in the implementations.
  
  
  ---
  
  