Answers.Body: <blockquote>
  <p>Now, let us assume the agent is in the penultimate state, <span class="math-container">$S_1$</span>, and
  chooses the action <span class="math-container">$A$</span> that leads him to the completion state, <span class="math-container">$S_2$</span>,
  and gets a reward <span class="math-container">$R$</span>.</p>
  <p><em>How do we form the target value <span class="math-container">$Q_\text{target}$</span> for <span class="math-container">$S_1$</span> now? Do we still include the <span class="math-container">$Q_\text{future}$</span> term? Or is it only the reward
  in this case?</em></p>
  </blockquote>
  <p>Your term &quot;completion state&quot; is commonly called &quot;terminal state&quot;. In a terminal state, there are no more actions to take, no more time steps, and no possibility to take any action. So, by definition, in your state <span class="math-container">$S_2$</span>, the expected future reward is <span class="math-container">$0$</span>.</p>
  <p>Mathematically, this is often noted like <span class="math-container">$v(S_T) = 0$</span> or <span class="math-container">$q(S_T,\cdot) = 0$</span> with the <span class="math-container">$T$</span> standing for last time step of the episode, and dot standing in for the fact that no action need to be supplied, or the specific action value is not relevant. So, therefore using your terms, <span class="math-container">$Q_\text{future} = \text{max}_a Q(S_2, a) = 0$</span></p>
  <p>That makes the equations work in theory, but does not explain what to do in <em>code</em>. In practice in your code, you would do as you suggest and use <em>just the reward</em> when calculating the TD target for <span class="math-container">$Q(S_1, A)$</span>. This is typically done using an <code>if</code> block around the <code>done</code> condition e.g.</p>
  <pre><code>if done:
    td_target = r
  else:
    td_target = r + gamma * np.max(q_future_values) 
  end
  </code></pre>
  <p>Of course, the details depend on how you have structured and named your variables. You will find code similar to this in most DQN implementations though.</p>
  <p>You should not really try to <em>learn</em> <span class="math-container">$V(S_2)$</span> or <span class="math-container">$Q(S_2, A)$</span>, or calculate TD target starting from <span class="math-container">$S_2$</span>, because the result should be <span class="math-container">$0$</span> by definition.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-15T17:39:56.637Z
Answers.LastActivityDate: 2020-11-01T14:37:14.337Z
Answers.LastEditDate: 2020-11-01T14:37:14.337Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am working on a deep reinforcement learning problem. Throughout the episode, there is a small positive and negative reward for good or bad decisions. In the end, there is a huge reward for the completion of the episode. So, this reward function is quite sparse.</p>
  <p>This is my understanding of how DQN works. The neural network predicts quality values for each possible action that can be taken from a state <span class="math-container">$S_1$</span>. Let us assume the predicted quality value for an action <span class="math-container">$A$</span> is <span class="math-container">$Q(S_1, A)$</span>, and this action allows the agent to reach <span class="math-container">$S_2$</span>.</p>
  <p>We now need the target quality value <span class="math-container">$Q_\text{target}$</span>, so that using <span class="math-container">$Q(S_1, A)$</span> and <span class="math-container">$Q_\text{target}$</span> the temporal difference can be calculated, and updates can be made to the parameters of the value network.</p>
  <p><span class="math-container">$Q_\text{target}$</span> is composed of two terms. The immediate reward <span class="math-container">$R$</span> and the maximum quality value of the resulting state that this chosen action leaves us in, which can be denoted by <span class="math-container">$Q_\text{future} = \text{max}_a Q(S_2, a)$</span>, which is in practice obtained by feeding the new state <span class="math-container">$S_2$</span> into the neural network and choosing (from the list of quality values for each action) the maximum quality value. We then multiply the discount factor <span class="math-container">$\gamma$</span> with this <span class="math-container">$Q_\text{future}$</span> and add it to the reward <span class="math-container">$R$</span>, i.e. <span class="math-container">$Q_\text{target} = R + \gamma \text{max}_a Q(S_2, a) = R + \gamma Q_\text{future}$</span>.</p>
  <p>Now, let us assume the agent is in the penultimate state, <span class="math-container">$S_1$</span>, and chooses the action <span class="math-container">$A$</span> that leads him to the completion state, <span class="math-container">$S_2$</span>, and gets a reward <span class="math-container">$R$</span>.</p>
  <p><em>How do we form the target value <span class="math-container">$Q_\text{target}$</span> for <span class="math-container">$S_1$</span> now? Do we still include the <span class="math-container">$Q_\text{future}$</span> term? Or is it only the reward in this case?</em> I am not sure if <span class="math-container">$Q_\text{future}$</span> even has meaning after reaching the final state <span class="math-container">$S_2$</span>. So, I think that, for the final step, the target value must simply be the reward. Is this right?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-01T14:21:38.390Z
Comments.Text: Although your question is put in the context of DQN and Q-learning, this could be closed as a duplicate of https://ai.stackexchange.com/q/3758/2444, which is asked in the context of SARSA: for this reason, I will actually not mark your post as a duplicate, although both questions have the same answer. [Here](https://ai.stackexchange.com/q/10329/2444) and [here](https://ai.stackexchange.com/q/22130/2444) are other related/similar/duplicate questions.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><q-learning><dqn><deep-rl>
Title: How do we compute the target value when the agent ends up in the terminal state?
UserDisplayName: pranav
ThreadText: # How do we compute the target value when the agent ends up in the terminal state?
  
  (Asked by: username_0 on 2019-10-15)
  
  I am working on a deep reinforcement learning problem. Throughout the episode, there is a small positive and negative reward for good or bad decisions. In the end, there is a huge reward for the completion of the episode. So, this reward function is quite sparse.
  
  This is my understanding of how DQN works. The neural network predicts quality values for each possible action that can be taken from a state $S\_1$. Let us assume the predicted quality value for an action $A$ is $Q(S\_1, A)$, and this action allows the agent to reach $S\_2$.
  
  We now need the target quality value $Q\_\text{target}$, so that using $Q(S\_1, A)$ and $Q\_\text{target}$ the temporal difference can be calculated, and updates can be made to the parameters of the value network.
  
  $Q\_\text{target}$ is composed of two terms. The immediate reward $R$ and the maximum quality value of the resulting state that this chosen action leaves us in, which can be denoted by $Q\_\text{future} = \text{max}\_a Q(S\_2, a)$, which is in practice obtained by feeding the new state $S\_2$ into the neural network and choosing (from the list of quality values for each action) the maximum quality value. We then multiply the discount factor $\gamma$ with this $Q\_\text{future}$ and add it to the reward $R$, i.e. $Q\_\text{target} = R + \gamma \text{max}\_a Q(S\_2, a) = R + \gamma Q\_\text{future}$.
  
  Now, let us assume the agent is in the penultimate state, $S\_1$, and chooses the action $A$ that leads him to the completion state, $S\_2$, and gets a reward $R$.
  
  *How do we form the target value $Q\_\text{target}$ for $S\_1$ now? Do we still include the $Q\_\text{future}$ term? Or is it only the reward in this case?* I am not sure if $Q\_\text{future}$ even has meaning after reaching the final state $S\_2$. So, I think that, for the final step, the target value must simply be the reward. Is this right?
  
  **Tags:** reinforcement-learning, q-learning, dqn, deep-rl
  > **Comments:**
  > **username_1**: Although your question is put in the context of DQN and Q-learning, this could be closed as a duplicate of https://ai.stackexchange.com/q/3758/2444, which is asked in the context of SARSA: for this reason, I will actually not mark your post as a duplicate, although both questions have the same answer. [Here](https://ai.stackexchange.com/q/10329/2444) and [here](https://ai.stackexchange.com/q/22130/2444) are other related/similar/duplicate questions.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-10-15. Score: 4 | Accepted answer
  
  Now, let us assume the agent is in the penultimate state, $S_1$, and
  chooses the action $A$ that leads him to the completion state, $S_2$,
  and gets a reward $R$.
  How do we form the target value $Q_\text{target}$ for $S_1$ now? Do we still include the $Q_\text{future}$ term? Or is it only the reward
  in this case?
  
  Your term "completion state" is commonly called "terminal state". In a terminal state, there are no more actions to take, no more time steps, and no possibility to take any action. So, by definition, in your state $S_2$, the expected future reward is $0$.
  Mathematically, this is often noted like $v(S_T) = 0$ or $q(S_T,\cdot) = 0$ with the $T$ standing for last time step of the episode, and dot standing in for the fact that no action need to be supplied, or the specific action value is not relevant. So, therefore using your terms, $Q_\text{future} = \text{max}_a Q(S_2, a) = 0$
  That makes the equations work in theory, but does not explain what to do in code. In practice in your code, you would do as you suggest and use just the reward when calculating the TD target for $Q(S_1, A)$. This is typically done using an if block around the done condition e.g.
  if done:
    td_target = r
  else:
    td_target = r + gamma * np.max(q_future_values) 
  end
  
  Of course, the details depend on how you have structured and named your variables. You will find code similar to this in most DQN implementations though.
  You should not really try to learn $V(S_2)$ or $Q(S_2, A)$, or calculate TD target starting from $S_2$, because the result should be $0$ by definition.
  
  
  ---
  
  