Answers.Body: <p>You are looking for the best actions which minimize the loss function. You sample a batch of memory buffer uniformly and define a loss function based on that batch. The memory buffer consists of trajectories. Each trajectory consists of an state and the action taken in that state which results in next state and an immediate reward. 
  If the trajectory is shown by <span class="math-container">$(s,a,r,s\prime)$</span>, the loss for this single state is simply defined as: <span class="math-container">$(r + max_a\prime Q(s\prime,a\prime,w^-)-Q(s,a,w))^2$</span>. </p>
  
  <p>The minus sign above the parameters means that you should fix the the target parameters to ensure the stability of learning. 
  So the loss function for a whole batch is:
  <span class="math-container">$L(w) = E_{(s,a,r,s\prime)\sim U(D)}(r + max_a\prime Q(s\prime,a\prime,w^-)-Q(s,a,w))^2$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-04T07:18:17.210Z
Answers.LastActivityDate: 2020-05-04T07:18:17.210Z
Answers.UserDisplayName: bitWise
Answers.Body: <blockquote>
    <p>When doing gradient descent update for this single example, should the target output to set for the network be equivalent to <span class="math-container">$Q(s_1,a_0), Q(s_1,a_1), r_2 + \gamma max_aQ(s',a',\theta) , Q(s_1,a_3),...$</span> ?</p>
  </blockquote>
  
  <p>Other than what looks like a couple of small typos, then yes.</p>
  
  <p>This is an implementation issue for DQN, where you have decided to create a function that outputs multiple Q functions at once. There is nothing about this in Q learning theory, so you need to figure out what will generate the correct error (and therefore gradients) for an update step. </p>
  
  <p>You don't know the TD targets for actions that were not taken, and cannot make any update for them, so the gradients for these actions must be zero. One way to achieve that is to feed back the network's own output for those actions. This is common practice because you can use built-in functions from neural network libraries to handle minibatches*. </p>
  
  <p>There are some details worth clarifying:</p>
  
  <ul>
  <li><p>You have substituted the third entry in the array with the calculated TD target because the action from experience replay is <span class="math-container">$a_2$</span>. In general you substitute for the action taken. Looks like you have this correct.</p></li>
  <li><p>You have <span class="math-container">$r_1$</span> in your experience replay table, but put <span class="math-container">$r_2$</span> in your TD target formula. Looks like a typo. Another typo is that you maximise over <span class="math-container">$a$</span> but reference <span class="math-container">$a'$</span>. Also, you reference <span class="math-container">$s'$</span> but don't define it anywhere. Fixing these issues gives <span class="math-container">$r_1 + \gamma \text{max}_{a'}Q(s_2,a',\theta)$</span></p></li>
  <li><p>For the TD target it is often worth using a dedicated <em>target</em> network that every N steps is copied from the learning network. It helps with stability. This can be noted as a "frozen copy" of <span class="math-container">$\theta$</span> noted <span class="math-container">$\theta^-$</span>, and the neural network approximate Q function often noted <span class="math-container">$\hat{q}$</span> giving formula of <span class="math-container">$r_1 + \gamma \text{max}_{a'}\hat{q}(s_2,a',\theta^-)$</span> for your example.</p></li>
  </ul>
  
  <hr>
  
  <p>* If you want you can also calculate the gradient more directly from the single action that was taken, and back propagate from there, knowing that all the other outputs will have a gradient componnet of zero. That requires implementing at least some of the back propagation yourself.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-04T08:10:31.350Z
Answers.LastActivityDate: 2020-05-04T08:16:27.857Z
Answers.LastEditDate: 2020-05-04T08:16:27.857Z
Answers.UserDisplayName: Neil Slater
Body: <p>I have some gaps in my understanding regarding the performing of the gradient descent in Deep - Q networks. <a href="https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf" rel="nofollow noreferrer">The original deep q network for Atari</a> performs a gradient descent step to minimise <span class="math-container">$y_j - Q(s_j,a_j,\theta)$</span>, where <span class="math-container">$y_j = r_j + \gamma max_aQ(s',a',\theta)$</span>. </p>
  
  <p>In the example where I sample a single experience <span class="math-container">$(s_1,a_2,r_1,s_2)$</span> and I try to conduct a single gradient descent step, then feeding in <span class="math-container">$s_1$</span> to the neural network outputs an array of <span class="math-container">$Q(s_1,a_0), Q(s_1,a_1), Q(s_1,a_2), \dots$</span> values. </p>
  
  <p>When doing gradient descent update for this single example, should the target output to set for the network be equivalent to <span class="math-container">$Q(s_1,a_0), Q(s_1,a_1), r_1 + \gamma max_{a'}Q(s_2,a',\theta), Q(s_1,a_3), \dots$</span> ?</p>
  
  <p>I know the inputs to the neural network to be <span class="math-container">$s_j$</span>, to give the corresponding Q values. However, I cannot concretize the target values that the network should be optimized. </p>
  
Tags: <neural-networks><reinforcement-learning><dqn><deep-rl>
Title: What should the target be when the neural network outputs multiple Q values in deep Q-learning?
UserDisplayName: calveeen
ThreadText: # What should the target be when the neural network outputs multiple Q values in deep Q-learning?
  
  (Asked by: username_0 on 2020-05-04)
  
  I have some gaps in my understanding regarding the performing of the gradient descent in Deep - Q networks. [The original deep q network for Atari](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf) performs a gradient descent step to minimise $y\_j - Q(s\_j,a\_j,\theta)$, where $y\_j = r\_j + \gamma max\_aQ(s',a',\theta)$.
  
  In the example where I sample a single experience $(s\_1,a\_2,r\_1,s\_2)$ and I try to conduct a single gradient descent step, then feeding in $s\_1$ to the neural network outputs an array of $Q(s\_1,a\_0), Q(s\_1,a\_1), Q(s\_1,a\_2), \dots$ values.
  
  When doing gradient descent update for this single example, should the target output to set for the network be equivalent to $Q(s\_1,a\_0), Q(s\_1,a\_1), r\_1 + \gamma max\_{a'}Q(s\_2,a',\theta), Q(s\_1,a\_3), \dots$ ?
  
  I know the inputs to the neural network to be $s\_j$, to give the corresponding Q values. However, I cannot concretize the target values that the network should be optimized.
  
  **Tags:** neural-networks, reinforcement-learning, dqn, deep-rl
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-05-04. Score: 3 | Accepted answer
  
  > When doing gradient descent update for this single example, should the target output to set for the network be equivalent to $Q(s\_1,a\_0), Q(s\_1,a\_1), r\_2 + \gamma max\_aQ(s',a',\theta) , Q(s\_1,a\_3),...$ ?
  
  Other than what looks like a couple of small typos, then yes.
  
  This is an implementation issue for DQN, where you have decided to create a function that outputs multiple Q functions at once. There is nothing about this in Q learning theory, so you need to figure out what will generate the correct error (and therefore gradients) for an update step.
  
  You don't know the TD targets for actions that were not taken, and cannot make any update for them, so the gradients for these actions must be zero. One way to achieve that is to feed back the network's own output for those actions. This is common practice because you can use built-in functions from neural network libraries to handle minibatches\*.
  
  There are some details worth clarifying:
  
  * You have substituted the third entry in the array with the calculated TD target because the action from experience replay is $a\_2$. In general you substitute for the action taken. Looks like you have this correct.
  * You have $r\_1$ in your experience replay table, but put $r\_2$ in your TD target formula. Looks like a typo. Another typo is that you maximise over $a$ but reference $a'$. Also, you reference $s'$ but don't define it anywhere. Fixing these issues gives $r\_1 + \gamma \text{max}\_{a'}Q(s\_2,a',\theta)$
  * For the TD target it is often worth using a dedicated *target* network that every N steps is copied from the learning network. It helps with stability. This can be noted as a "frozen copy" of $\theta$ noted $\theta^-$, and the neural network approximate Q function often noted $\hat{q}$ giving formula of $r\_1 + \gamma \text{max}\_{a'}\hat{q}(s\_2,a',\theta^-)$ for your example.
  
  ---
  
  \* If you want you can also calculate the gradient more directly from the single action that was taken, and back propagate from there, knowing that all the other outputs will have a gradient componnet of zero. That requires implementing at least some of the back propagation yourself.
  
  
  ---
  
  ### Answer by username_1 on 2020-05-04. Score: 1
  
  You are looking for the best actions which minimize the loss function. You sample a batch of memory buffer uniformly and define a loss function based on that batch. The memory buffer consists of trajectories. Each trajectory consists of an state and the action taken in that state which results in next state and an immediate reward.
  If the trajectory is shown by $(s,a,r,s\prime)$, the loss for this single state is simply defined as: $(r + max\_a\prime Q(s\prime,a\prime,w^-)-Q(s,a,w))^2$.
  
  The minus sign above the parameters means that you should fix the the target parameters to ensure the stability of learning.
  So the loss function for a whole batch is:
  $L(w) = E\_{(s,a,r,s\prime)\sim U(D)}(r + max\_a\prime Q(s\prime,a\prime,w^-)-Q(s,a,w))^2$.
  
  
  ---
  
  