Answers.Body: <blockquote>
  <p>When we use our network to approximate our Q values,is the Q target a single value?</p>
  </blockquote>
  <p>Yes, the target Q value is a single value if you are just updating a single training example. The loss function of a vanilla DQN for a single experience tuple <span class="math-container">$(s_t,a_t,r_t,s_{t+1})$</span> is calculated as <span class="math-container">$$L(\theta) = [r_t + \gamma \,max\,Q_{a_{t+1}}(s_{t+1},a_{t+1};\theta) - Q(s_t,a_t;\theta)]^2$$</span> where <span class="math-container">$r_t + \gamma \,max\,Q(s_{t+1},a_{t+1};\theta)$</span> is the target Q value. However, when using mini-batch gradient descent, you would have to compute multiple target Q values equivalent to the batch size</p>
  <blockquote>
  <p>During backpropagation, when the weights are updated, does it automatically update the Q values, shouldn’t the state be passed in the network again to update it?</p>
  </blockquote>
  <p>During backpropagation of the loss function, the weights <span class="math-container">$\theta$</span> are automatically updated. You do not need to pass in the state again. Because in the first place, you would have computed <span class="math-container">$Q(s_t,a_t;\theta)$</span> by passing in the state as input to the neural network. That is how backpropagation works for Deep Q networks.</p>
  <p>Training for the DQN is as follows:</p>
  <ol>
  <li>Collect experience tuples of <span class="math-container">$(s_t,a_t,r_t,s_{t+1})$</span> and store them in a replay buffer.</li>
  <li>Sample mini-batch of experiences from the replay buffer.</li>
  <li>From these sampled batch of experiences, compute <span class="math-container">$Q(s_{t+1}.a_{t+1};\theta)$</span> by passing <span class="math-container">$s_{t+1}$</span> into the network and take the Q value with the maximum values</li>
  <li>Compute <span class="math-container">$Q(s_t,a_t;\theta)$</span> by passing <span class="math-container">$s_t$</span> into the network.</li>
  <li>Compute the Loss for this experience and propagate the loss back to the network, hence updating the weights.</li>
  </ol>
  <p>Also, since the weights have changed after backpropagation, the Q values for the same state would also be updated if you pass the same state in to the network again. <a href="https://www.cs.toronto.edu/%7Evmnih/docs/dqn.pdf" rel="nofollow noreferrer">Check out this paper as it explains how Deep Q Network works</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-12T22:47:54.987Z
Answers.Text: @Chukwudi You are very confused. I suggest that you first learn about **tabular** Q-learning before getting into deep Q-learning! The neural network $M_{\theta_i}$ represents your Q function at time step $i$. If you update the parameters of this neural network, you also change your Q function. So, after an update of the parameters of this network that represents the Q function, then $M_{\theta_{i+1}}(s)$ will (probably) be different than $M_{\theta_i}(s)$, where $\theta_i$ are the parameters of the net _before_ the update.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-13T10:35:55.497Z
Answers.Text: I understand the table, I just realized that it’s a function approximation, so anytime the weights change the Q clause also change to move closer to the respective Q state action pair values , and we meet the weights of our target network constant to introduce stability, David Ireland explained it in detail
Answers.UserDisplayName: Chukwudi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-12T16:30:54.423Z
Answers.LastActivityDate: 2020-06-12T16:39:16.187Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: calveeen
Answers.Body: <p>What the network represents can be a little confusing - in tabular Q-learning you have a <span class="math-container">$Q$</span> function that you pass a state <span class="math-container">$s$</span> and action <span class="math-container">$a$</span> into and receive a scalar value. In the <a href="https://www.nature.com/articles/nature14236" rel="nofollow noreferrer">Human Level Control</a> paper where the DQN gained its popularity, the network is a little different to the tabular function. You pass into the network your current state and it outputs values for every action in your action space. You then choose the action which has the highest value. </p>
  
  <p>The way we then train it is as follows - first we get our scalar value which corresponds to the target <span class="math-container">$r + \gamma max_{a'} Q(s',a')$</span>; I will denote this as <span class="math-container">$y$</span> for short. As our network outputs a vector of dimension <span class="math-container">$|\mathcal{A}|$</span> where <span class="math-container">$\mathcal{A}$</span> is our action space, we get our prediction for the current state which is a vector of the dimension I just mentioned. We will now assume our action space has two possible actions and say that for state <span class="math-container">$s$</span> our network outputs the vector <span class="math-container">$[\hat{y}_1,\hat{y}_2]$</span>. Assume that when we calculated <span class="math-container">$y$</span> earlier we took the first action, so we only want to update our prediction of <span class="math-container">$\hat{y}_1$</span>. To do this, if we were working in e.g. PyTorch I would put into the loss function, which is mean squared error, </p>
  
  <blockquote>
    <p>input = <span class="math-container">$[\hat{y}_1,\hat{y}_2$</span>], target = <span class="math-container">$[y, \hat{y}_2]$</span></p>
  </blockquote>
  
  <p>so essentially all we change is the position in the vector corresponding to the action we took with the Q-target we observed. The weights will thus be updated accordingly to move more into the direction of producing this output for state <span class="math-container">$s$</span>. </p>
  
  <p>So to explicitly answer your first question, the Q-target is a scalar but the target we pass into the network is a vector. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-12T22:18:37.243Z
Answers.Text: I think you are misunderstanding what happens in Q-learning. The network is a function of the state. Imagine a normal neural network that is a function of x, when you update the weights the next time you feed the same x in it will (probably) give you a different value. There’s not really anything else that can be said unless you explain further what you don’t understand.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-13T10:33:25.030Z
Answers.Text: Ohh I think I get it, it’s a function approximation, so as long as the weights change , the next time the state is fed the Q values also change, it doesn’t matter about changing immediately
Answers.UserDisplayName: Chukwudi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-12T16:33:52.587Z
Answers.LastActivityDate: 2020-06-12T16:33:52.587Z
Answers.UserDisplayName: David
Body: <p>I have two questions </p>
  
  <ol>
  <li><p>When we use our network to approximate our Q values, is the Q target a single value?</p></li>
  <li><p>During backpropagation, when the weights are updated, does it automatically update the Q values, shouldn’t the state be passed in the network again to update it?</p></li>
  </ol>
  
Tags: <neural-networks><reinforcement-learning><q-learning><dqn><deep-rl>
Title: When we use a neural network to approximate the Q values, is the Q target a single value?
UserDisplayName: Chukwudi
ThreadText: # When we use a neural network to approximate the Q values, is the Q target a single value?
  
  (Asked by: username_0 on 2020-06-12)
  
  I have two questions
  
  1. When we use our network to approximate our Q values, is the Q target a single value?
  2. During backpropagation, when the weights are updated, does it automatically update the Q values, shouldn’t the state be passed in the network again to update it?
  
  **Tags:** neural-networks, reinforcement-learning, q-learning, dqn, deep-rl
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-06-12. Score: 1 | Accepted answer
  
  > When we use our network to approximate our Q values,is the Q target a single value?
  
  Yes, the target Q value is a single value if you are just updating a single training example. The loss function of a vanilla DQN for a single experience tuple $(s\_t,a\_t,r\_t,s\_{t+1})$ is calculated as $$L(\theta) = [r\_t + \gamma \,max\,Q\_{a\_{t+1}}(s\_{t+1},a\_{t+1};\theta) - Q(s\_t,a\_t;\theta)]^2$$ where $r\_t + \gamma \,max\,Q(s\_{t+1},a\_{t+1};\theta)$ is the target Q value. However, when using mini-batch gradient descent, you would have to compute multiple target Q values equivalent to the batch size
  
  > During backpropagation, when the weights are updated, does it automatically update the Q values, shouldn’t the state be passed in the network again to update it?
  
  During backpropagation of the loss function, the weights $\theta$ are automatically updated. You do not need to pass in the state again. Because in the first place, you would have computed $Q(s\_t,a\_t;\theta)$ by passing in the state as input to the neural network. That is how backpropagation works for Deep Q networks.
  
  Training for the DQN is as follows:
  
  1. Collect experience tuples of $(s\_t,a\_t,r\_t,s\_{t+1})$ and store them in a replay buffer.
  2. Sample mini-batch of experiences from the replay buffer.
  3. From these sampled batch of experiences, compute $Q(s\_{t+1}.a\_{t+1};\theta)$ by passing $s\_{t+1}$ into the network and take the Q value with the maximum values
  4. Compute $Q(s\_t,a\_t;\theta)$ by passing $s\_t$ into the network.
  5. Compute the Loss for this experience and propagate the loss back to the network, hence updating the weights.
  
  Also, since the weights have changed after backpropagation, the Q values for the same state would also be updated if you pass the same state in to the network again. [Check out this paper as it explains how Deep Q Network works](https://www.cs.toronto.edu/%7Evmnih/docs/dqn.pdf).
  
  > **Comments:**
  > **username_2**: @username_0 You are very confused. I suggest that you first learn about **tabular** Q-learning before getting into deep Q-learning! The neural network $M_{\theta_i}$ represents your Q function at time step $i$. If you update the parameters of this neural network, you also change your Q function. So, after an update of the parameters of this network that represents the Q function, then $M_{\theta_{i+1}}(s)$ will (probably) be different than $M_{\theta_i}(s)$, where $\theta_i$ are the parameters of the net _before_ the update.  
  > **username_0**: I understand the table, I just realized that it’s a function approximation, so anytime the weights change the Q clause also change to move closer to the respective Q state action pair values , and we meet the weights of our target network constant to introduce stability, username_3 Ireland explained it in detail  
  
  ---
  
  ### Answer by username_3 on 2020-06-12. Score: 0
  
  What the network represents can be a little confusing - in tabular Q-learning you have a $Q$ function that you pass a state $s$ and action $a$ into and receive a scalar value. In the [Human Level Control](https://www.nature.com/articles/nature14236) paper where the DQN gained its popularity, the network is a little different to the tabular function. You pass into the network your current state and it outputs values for every action in your action space. You then choose the action which has the highest value.
  
  The way we then train it is as follows - first we get our scalar value which corresponds to the target $r + \gamma max\_{a'} Q(s',a')$; I will denote this as $y$ for short. As our network outputs a vector of dimension $|\mathcal{A}|$ where $\mathcal{A}$ is our action space, we get our prediction for the current state which is a vector of the dimension I just mentioned. We will now assume our action space has two possible actions and say that for state $s$ our network outputs the vector $[\hat{y}\_1,\hat{y}\_2]$. Assume that when we calculated $y$ earlier we took the first action, so we only want to update our prediction of $\hat{y}\_1$. To do this, if we were working in e.g. PyTorch I would put into the loss function, which is mean squared error,
  
  > input = $[\hat{y}\_1,\hat{y}\_2$], target = $[y, \hat{y}\_2]$
  
  so essentially all we change is the position in the vector corresponding to the action we took with the Q-target we observed. The weights will thus be updated accordingly to move more into the direction of producing this output for state $s$.
  
  So to explicitly answer your first question, the Q-target is a scalar but the target we pass into the network is a vector.
  
  > **Comments:**
  > **username_3**: I think you are misunderstanding what happens in Q-learning. The network is a function of the state. Imagine a normal neural network that is a function of x, when you update the weights the next time you feed the same x in it will (probably) give you a different value. There’s not really anything else that can be said unless you explain further what you don’t understand.  
  > **username_0**: Ohh I think I get it, it’s a function approximation, so as long as the weights change , the next time the state is fed the Q values also change, it doesn’t matter about changing immediately  
  
  ---
  
  