Answers.Body: <p>This is the problem that reinforcement learning (RL) is trying to solve: What is the best way to behave when we don’t know what the right action is and only have a scalar (the reward (r) is a scalar) reward of how well we have done?</p>
  
  <p>RL approaches this problem by utilizing temporal difference learning and makes predictions based on the previous experience. An RL agent is trying to maximize the sum of future discounted rewards, called the return. </p>
  
  <p>The term <span class="math-container">$r + \gamma \max_{a'} Q(s', a'; \theta_i^-)$</span> is essentially saying "<em>the reward I just saw + <span class="math-container">$\gamma$</span> * (my prediction of the return given that I take what I think is the best action in the current state and follow my policy from then on)"</em>. </p>
  
  <p>By updating the Q function, the agent can better predict the consequences of its actions and can then choose the best action with greater probability. </p>
  
  <p>The <span class="math-container">$\gamma$</span> (gamma) helps to balance between the immediate reward and future rewards. A <span class="math-container">$\gamma = 0$</span> makes the immediate reward the only thing that matters but usually the reward for a good action is delayed so values of gamma which put a higher importance on later rewards (<span class="math-container">$\gamma = .8, .9, .99$</span>, etc) are used. </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-14T22:45:58.547Z
Answers.Text: my prediction of the return ***given that I take what I think is the best action in the current state and follow my policy (theta) from then on*** why do we need this disclaimer? Why can't it just be the predicted return?
Answers.UserDisplayName: echo
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-14T22:52:12.107Z
Answers.Text: Because we want the return to be based on our current policy so that we can improve our policy.
Answers.UserDisplayName: Jaden Travnik
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-14T23:11:56.257Z
Answers.Text: That doesn't make sense to me. We want the return to have a recursive formulation? And by having a recursive formulation, we can improve our policy?
Answers.UserDisplayName: echo
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-14T23:57:54.367Z
Answers.Text: It’s recursive yes because it’s based on the previous steps.  Reading chapter 6 of the Introduction to RL book by Sutton and Barto should help. http://ufal.mff.cuni.cz/~straka/courses/npfl114/2016/sutton-bookdraft2016sep.pdf
Answers.UserDisplayName: Jaden Travnik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-12-14T16:11:57.743Z
Answers.LastActivityDate: 2018-11-19T07:37:07.087Z
Answers.LastEditDate: 2018-11-19T07:37:07.087Z
Answers.UserDisplayName: Jaden Travnik
Body: <p>In the paper <a href="https://www.nature.com/articles/nature14236" rel="nofollow noreferrer">Human-level control through deep reinforcement learning</a>, the DQN architecture is presented, where the loss function is as follows</p>
  
  <p><span class="math-container">$$
  L_i(\theta_i) = \mathbb{E}_{(s, a, r, s') \sim U(D)} \left[ \left( r + \gamma \max_{a'} Q(s', a'; \theta_i^-)  - Q(s, a; \theta) \right)^2\right]
  $$</span></p>
  
  <p>where <span class="math-container">$r + \gamma \max_{a'} Q(s', a'; \theta_i^-)$</span> approximates the "target" of <span class="math-container">$Q(s, a; \theta)$</span>. But it is not clear to me why. How can existing weights approximate the target (ground truth)? Isn't <span class="math-container">$r$</span> is a sample from a the experience replay dataset? Is <span class="math-container">$r$</span> a scalar value?</p>
  
Tags: <deep-learning><reinforcement-learning><q-learning>
Title: Why is the target $r + \gamma \max_{a'} Q(s', a'; \theta_i^-)$ in the loss function of the DQN architecture?
UserDisplayName: echo
ThreadText: # Why is the target $r + \gamma \max_{a'} Q(s', a'; \theta_i^-)$ in the loss function of the DQN architecture?
  
  (Asked by: username_0 on 2017-12-13)
  
  In the paper [Human-level control through deep reinforcement learning](https://www.nature.com/articles/nature14236), the DQN architecture is presented, where the loss function is as follows
  
  $$
  L\_i(\theta\_i) = \mathbb{E}\_{(s, a, r, s') \sim U(D)} \left[ \left( r + \gamma \max\_{a'} Q(s', a'; \theta\_i^-) - Q(s, a; \theta) \right)^2\right]
  $$
  
  where $r + \gamma \max\_{a'} Q(s', a'; \theta\_i^-)$ approximates the "target" of $Q(s, a; \theta)$. But it is not clear to me why. How can existing weights approximate the target (ground truth)? Isn't $r$ is a sample from a the experience replay dataset? Is $r$ a scalar value?
  
  **Tags:** deep-learning, reinforcement-learning, q-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2017-12-14. Score: 6 | Accepted answer
  
  This is the problem that reinforcement learning (RL) is trying to solve: What is the best way to behave when we don’t know what the right action is and only have a scalar (the reward (r) is a scalar) reward of how well we have done?
  
  RL approaches this problem by utilizing temporal difference learning and makes predictions based on the previous experience. An RL agent is trying to maximize the sum of future discounted rewards, called the return.
  
  The term $r + \gamma \max\_{a'} Q(s', a'; \theta\_i^-)$ is essentially saying "*the reward I just saw + $\gamma$ \* (my prediction of the return given that I take what I think is the best action in the current state and follow my policy from then on)"*.
  
  By updating the Q function, the agent can better predict the consequences of its actions and can then choose the best action with greater probability.
  
  The $\gamma$ (gamma) helps to balance between the immediate reward and future rewards. A $\gamma = 0$ makes the immediate reward the only thing that matters but usually the reward for a good action is delayed so values of gamma which put a higher importance on later rewards ($\gamma = .8, .9, .99$, etc) are used.
  
  > **Comments:**
  > **username_0**: my prediction of the return ***given that I take what I think is the best action in the current state and follow my policy (theta) from then on*** why do we need this disclaimer? Why can't it just be the predicted return?  
  > **username_1**: Because we want the return to be based on our current policy so that we can improve our policy.  
  > **username_0**: That doesn't make sense to me. We want the return to have a recursive formulation? And by having a recursive formulation, we can improve our policy?  
  > **username_1**: It’s recursive yes because it’s based on the previous steps.  Reading chapter 6 of the Introduction to RL book by Sutton and Barto should help. http://ufal.mff.cuni.cz/~straka/courses/npfl114/2016/sutton-bookdraft2016sep.pdf  
  
  ---
  
  