Answers.Body: <blockquote>
  <p>However, all this is doing is updating the parameters of the Neural Network to produce an output that matches the Target Q values as close as possible.</p>
  </blockquote>
  <p>Yes. That is all it needs to do because we have defined the policy around the Q values like so:</p>
  <p><span class="math-container">$$\pi(s) = \text{argmax}_a \hat{q}(s,a,\theta)$$</span></p>
  <p>Where <span class="math-container">$\theta$</span> is the neural network weights.</p>
  <p>Therefore, if the estimates of Q are approximately the same as the action value of the optimal policy, the policy in DQN is approximately the optimal policy.</p>
  <blockquote>
  <p>How does the algorithm know +10 is good?</p>
  </blockquote>
  <p>It does not, at least not directly. The algorithm knows, approximately, what the action values are <strong>if</strong> it acts consistently with their current estimates by always choosing the maximising action at each step.</p>
  <p>The learning process will learn that +10 is <em>relatively</em> good in your scenario because it never finds anything better when exploring.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-13T07:47:05.433Z
Answers.LastActivityDate: 2020-08-13T07:47:05.433Z
Answers.UserDisplayName: Neil Slater
Body: <p>I'm aware that we back-propagate after computing the loss between:</p>
  <p>The Neural Network Q values and the Target Network Q values</p>
  <p>However, all this is doing is updating the parameters of the Neural Network to produce an output that matches the Target Q values as close as possible.</p>
  <p>Suppose one epoch is run and the reward is +10, surely we need to update the parameters using this too to tell the Network to push the probability of these actions, given these parameters up.</p>
  <p>How does the algorithm know +10 is good? Suppose the reward range is -10 for loss and +10 for win.</p>
  
Tags: <neural-networks><machine-learning><reinforcement-learning><deep-learning><dqn>
Title: In DQN, when do the parameters in the Neural Network update based on the reward received?
UserDisplayName: THAT_AI_GUY
ThreadText: # In DQN, when do the parameters in the Neural Network update based on the reward received?
  
  (Asked by: username_0 on 2020-08-12)
  
  I'm aware that we back-propagate after computing the loss between:
  
  The Neural Network Q values and the Target Network Q values
  
  However, all this is doing is updating the parameters of the Neural Network to produce an output that matches the Target Q values as close as possible.
  
  Suppose one epoch is run and the reward is +10, surely we need to update the parameters using this too to tell the Network to push the probability of these actions, given these parameters up.
  
  How does the algorithm know +10 is good? Suppose the reward range is -10 for loss and +10 for win.
  
  **Tags:** neural-networks, machine-learning, reinforcement-learning, deep-learning, dqn
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-08-13. Score: 1 | Accepted answer
  
  > However, all this is doing is updating the parameters of the Neural Network to produce an output that matches the Target Q values as close as possible.
  
  Yes. That is all it needs to do because we have defined the policy around the Q values like so:
  
  $$\pi(s) = \text{argmax}\_a \hat{q}(s,a,\theta)$$
  
  Where $\theta$ is the neural network weights.
  
  Therefore, if the estimates of Q are approximately the same as the action value of the optimal policy, the policy in DQN is approximately the optimal policy.
  
  > How does the algorithm know +10 is good?
  
  It does not, at least not directly. The algorithm knows, approximately, what the action values are **if** it acts consistently with their current estimates by always choosing the maximising action at each step.
  
  The learning process will learn that +10 is *relatively* good in your scenario because it never finds anything better when exploring.
  
  
  ---
  
  