Answers.Body: <blockquote>
  <ol>
  <li>Intuitively, how does it make sense to take an action A' when the environment already ended?</li>
  </ol>
  </blockquote>
  <p>It doesn't make sense, in that nothing can happen once the agent reaches a terminal state. However, it is often modelled as an &quot;absorbing state&quot; where the action is unimportant (either <code>null</code> or value ignored) with value by definition of <span class="math-container">$0$</span>.</p>
  <blockquote>
  <p>And you can't set all the terminal values to zero because you are using function approximators</p>
  </blockquote>
  <p>The value is zero by definition. There is no need to approximate it. So don't use function approximators for action values in terminal states.  When <span class="math-container">$S'$</span> is terminal, the update becomes:</p>
  <p><span class="math-container">$Q(S,A) \leftarrow Q(S,A) + \alpha(R - Q(S,A))$</span></p>
  <p>Look at any implementation of Q learning and you will see a conditional calculation for the update value, that uses some variant of the above logic when <span class="math-container">$S'$</span> is terminal. For OpenAI Gym environments for instance, it will use the <code>done</code> flag.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-06T17:13:31.810Z
Answers.LastActivityDate: 2020-11-06T17:13:31.810Z
Answers.UserDisplayName: Neil Slater
Body: <p>The update equation for SARSA is <span class="math-container">$Q(S,A) = R + \gamma Q(S',A')$</span>. Consider this: I take an action <span class="math-container">$A$</span> that leads to the terminal state. Now my <span class="math-container">$S'$</span> would be one of the terminal states. So...</p>
  <ol>
  <li><p>Intuitively, how does it make sense to take an action <span class="math-container">$A'$</span> when the environment already ended? Or is this something you just do anyway?</p>
  </li>
  <li><p>Once a terminal state-action pair is reached, you update the previous state-action pair and then start the game loop all over again. But this means that the terminal state-action pair (<span class="math-container">$Q(S',A')$</span> in my example) is never updated. So, if your initial estimate of <span class="math-container">$Q(S',A')$</span> was wrong, you would never be able to fix it which would be very problematic. (And you can't set all the terminal values to zero because you are using function approximators)</p>
  </li>
  </ol>
  <p>So, how do I resolve these issues?</p>
  
Tags: <reinforcement-learning><sarsa>
Title: Intuitively, how does it make sense to take an action $A'$ when the environment already ended?
UserDisplayName: BOSSrobot
ThreadText: # Intuitively, how does it make sense to take an action $A'$ when the environment already ended?
  
  (Asked by: username_0 on 2020-11-06)
  
  The update equation for SARSA is $Q(S,A) = R + \gamma Q(S',A')$. Consider this: I take an action $A$ that leads to the terminal state. Now my $S'$ would be one of the terminal states. So...
  
  1. Intuitively, how does it make sense to take an action $A'$ when the environment already ended? Or is this something you just do anyway?
  2. Once a terminal state-action pair is reached, you update the previous state-action pair and then start the game loop all over again. But this means that the terminal state-action pair ($Q(S',A')$ in my example) is never updated. So, if your initial estimate of $Q(S',A')$ was wrong, you would never be able to fix it which would be very problematic. (And you can't set all the terminal values to zero because you are using function approximators)
  
  So, how do I resolve these issues?
  
  **Tags:** reinforcement-learning, sarsa
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-11-06. Score: 1 | Accepted answer
  
  > 1. Intuitively, how does it make sense to take an action A' when the environment already ended?
  
  It doesn't make sense, in that nothing can happen once the agent reaches a terminal state. However, it is often modelled as an "absorbing state" where the action is unimportant (either `null` or value ignored) with value by definition of $0$.
  
  > And you can't set all the terminal values to zero because you are using function approximators
  
  The value is zero by definition. There is no need to approximate it. So don't use function approximators for action values in terminal states. When $S'$ is terminal, the update becomes:
  
  $Q(S,A) \leftarrow Q(S,A) + \alpha(R - Q(S,A))$
  
  Look at any implementation of Q learning and you will see a conditional calculation for the update value, that uses some variant of the above logic when $S'$ is terminal. For OpenAI Gym environments for instance, it will use the `done` flag.
  
  
  ---
  
  