Answers.Body: <p>It is quite common in DQN to instead of having the neural network represent function <span class="math-container">$f(s,a) = \hat{q}(s,a,\theta)$</span> directly, it actually represents <span class="math-container">$f(s)= [\hat{q}(s,1,\theta), \hat{q}(s,2,\theta), \hat{q}(s,3,\theta) . . . \hat{q}(s,N_a,\theta)]$</span> where <span class="math-container">$N_a$</span> is the maximum action, and the input the current state. That is what is going on here. It is usually done for a performance gain, since calculating all values at once is faster than individually.</p>
  
  <p>However, in a Q learning update, you cannot adjust this vector of output values for  actions that you did not take. You can do one of two things:</p>
  
  <ul>
  <li><p>Figure out the gradient due to the one item with a TD error, and propagate that backwards. This involves inserting a known gradient into the normal training update step in a specific place and working from there. This works best if you are implementing your own backpropagation with low-level tools, otherwise it can be a bit fiddly figuring out how to do it in a framework like Keras.</p></li>
  <li><p>Force the gradients of all other items to be zero by setting the target outputs to be whatever the learning network is currently generating.</p></li>
  </ul>
  
  <p>If you are using something like Keras, the second approach is the way to go. A concrete example where you have two networks <code>n_learn</code> and <code>n_target</code> that output arrays of Q values might be like this:</p>
  
  <ul>
  <li><p>For each sample <code>(s, a, r, next_s, done)</code> in your minibatch*</p>
  
  <ul>
  <li>Calculate array of action values from your learning network <code>qvals = n_learn.predict(s)</code></li>
  <li>Calulate TD target for <span class="math-container">$(s,a)$</span> e.g. <code>td_target = r + max(n_target.predict(next_s))</code> (discount factor and how to handle terminal states not shown)</li>
  <li>Alter the one array item that you know about from this sample <code>qvals[a] = td_target</code></li>
  <li>Append <code>s</code> to your <code>train_X</code> data and <code>qvals</code> to your <code>train_Y</code> data</li>
  </ul></li>
  <li><p>Fit the minibatch <code>n_learn.fit(train_X, train_Y)</code></p></li>
  </ul>
  
  <hr>
  
  <p>* It is possible to vectorise these calculations for efficiency. I show it as a for loop as it is simpler to describe that way</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-15T14:46:48.293Z
Answers.Text: Thank you again for answering. I think I got that. I also think that I implemented that the right way. Would you mind to look over it and tell my what I got wrong here?
  https://github.com/OleVoss/DeepTaxi
  
  PS:
  I tried it with various architectures and hyperparamaters.
Answers.UserDisplayName: Voß
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-15T15:46:41.843Z
Answers.Text: Looks ok on a first pass through, but I'm not familiar enough with PyTorch to comment on how you manipulated the q values in vectorised form. There are a few things you could try, including simplifying your network right down to no hidden layers, because essentially it is learning a Q table when each state is one hot encoded, and the weights will actually be equal to the learned Q values. There is no generalisation possible between states due to the way you have encoded them.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-15T16:05:45.743Z
Answers.Text: Ok. Appreciate your try. Technically it is learning something. But every time it is lerning to take just one action...
Answers.UserDisplayName: Voß
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-15T14:05:17.537Z
Answers.LastActivityDate: 2020-01-15T14:16:38.860Z
Answers.LastEditDate: 2020-01-15T14:16:38.860Z
Answers.UserDisplayName: Neil Slater
Body: <p>I'm currently trying to take the next step in deep learning. I managed so far to write my own basic feed-forward network in python without any frameworks (just numpy and pandas), so I think I understood the math and intuition behind backpropagation. Now, I'm stuck with deep q-learning. I've tried to get an agent to learn in various environments. But somehow nothing works out. So there has to be something I'm getting wrong. And it seems that I do not understand the critical part right at least that's what I'm thinking. <a href="https://i.sstatic.net/yIYj9.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/yIYj9.png" alt="enter image description here"></a></p>
  
  <p>The screenshot is from <a href="https://www.youtube.com/watch?v=NLaiMxndXKc" rel="nofollow noreferrer">this</a> video.</p>
  
  <p>What I'm trying to draw here is my understanding of the very basic process of a simple DQN. Assuming this is right, how is the loss backpropagated? 
  Since only the selected <span class="math-container">$Q(s, a)$</span> values (5 and 7) are further processed in the loss function, how is the impact from the other neurons calculated so their weights can be adjusted to better predict the real q-values?</p>
  
Tags: <deep-learning><reinforcement-learning><q-learning><backpropagation><dqn>
Title: How can a DQN backpropagate its loss?
UserDisplayName: Voß
ThreadText: # How can a DQN backpropagate its loss?
  
  (Asked by: username_0 on 2020-01-14)
  
  I'm currently trying to take the next step in deep learning. I managed so far to write my own basic feed-forward network in python without any frameworks (just numpy and pandas), so I think I understood the math and intuition behind backpropagation. Now, I'm stuck with deep q-learning. I've tried to get an agent to learn in various environments. But somehow nothing works out. So there has to be something I'm getting wrong. And it seems that I do not understand the critical part right at least that's what I'm thinking. [![enter image description here](https://i.sstatic.net/yIYj9.png)](https://i.sstatic.net/yIYj9.png)
  
  The screenshot is from [this](https://www.youtube.com/watch?v=NLaiMxndXKc) video.
  
  What I'm trying to draw here is my understanding of the very basic process of a simple DQN. Assuming this is right, how is the loss backpropagated?
  Since only the selected $Q(s, a)$ values (5 and 7) are further processed in the loss function, how is the impact from the other neurons calculated so their weights can be adjusted to better predict the real q-values?
  
  **Tags:** deep-learning, reinforcement-learning, q-learning, backpropagation, dqn
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-01-15. Score: 2 | Accepted answer
  
  It is quite common in DQN to instead of having the neural network represent function $f(s,a) = \hat{q}(s,a,\theta)$ directly, it actually represents $f(s)= [\hat{q}(s,1,\theta), \hat{q}(s,2,\theta), \hat{q}(s,3,\theta) . . . \hat{q}(s,N\_a,\theta)]$ where $N\_a$ is the maximum action, and the input the current state. That is what is going on here. It is usually done for a performance gain, since calculating all values at once is faster than individually.
  
  However, in a Q learning update, you cannot adjust this vector of output values for actions that you did not take. You can do one of two things:
  
  * Figure out the gradient due to the one item with a TD error, and propagate that backwards. This involves inserting a known gradient into the normal training update step in a specific place and working from there. This works best if you are implementing your own backpropagation with low-level tools, otherwise it can be a bit fiddly figuring out how to do it in a framework like Keras.
  * Force the gradients of all other items to be zero by setting the target outputs to be whatever the learning network is currently generating.
  
  If you are using something like Keras, the second approach is the way to go. A concrete example where you have two networks `n_learn` and `n_target` that output arrays of Q values might be like this:
  
  * For each sample `(s, a, r, next_s, done)` in your minibatch\*
  
    + Calculate array of action values from your learning network `qvals = n_learn.predict(s)`
    + Calulate TD target for $(s,a)$ e.g. `td_target = r + max(n_target.predict(next_s))` (discount factor and how to handle terminal states not shown)
    + Alter the one array item that you know about from this sample `qvals[a] = td_target`
    + Append `s` to your `train_X` data and `qvals` to your `train_Y` data
  * Fit the minibatch `n_learn.fit(train_X, train_Y)`
  
  ---
  
  \* It is possible to vectorise these calculations for efficiency. I show it as a for loop as it is simpler to describe that way
  
  > **Comments:**
  > **username_0**: Thank you again for answering. I think I got that. I also think that I implemented that the right way. Would you mind to look over it and tell my what I got wrong here? https://github.com/OleVoss/DeepTaxi  PS: I tried it with various architectures and hyperparamaters.  
  > **username_1**: Looks ok on a first pass through, but I'm not familiar enough with PyTorch to comment on how you manipulated the q values in vectorised form. There are a few things you could try, including simplifying your network right down to no hidden layers, because essentially it is learning a Q table when each state is one hot encoded, and the weights will actually be equal to the learned Q values. There is no generalisation possible between states due to the way you have encoded them.  
  > **username_0**: Ok. Appreciate your try. Technically it is learning something. But every time it is lerning to take just one action...  
  
  ---
  
  