Answers.Body: <p>From the description of the algorithm you linked to, it says to 'repeat until s is terminal'. So one would end the episode at that point and your intuition holds. </p>
  
  <p>Practically, if one was implementing a reward function where a specific reward is associated with the end of the episode such as "r(robot ran into a wall) = -100" then one can imagine that there is a terminal state just after this 'wall hit' state so the agent could see this reward. The episode would then be at a terminal state so would end. </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-05T10:24:12.977Z
Answers.LastActivityDate: 2017-08-05T10:24:12.977Z
Answers.UserDisplayName: Jaden Travnik
Answers.Body: <p>The value $Q(s', ~\cdot~)$ should always be implemented to simply be equal to $0$ for any terminal state $s'$ (the dot instead of an action as second argument there indicates that what I just wrote should hold for <strong>any action</strong>, as long as $s'$ is terminal).</p>
  
  <p>It is easier to understand why this should be the case by dissecting what the different terms in the update rule mean:</p>
  
  <p>$$Q(s, a) \gets \color{red}{Q(s, a)} + \alpha \left[ \color{blue}{r + \gamma Q(s', a')} - \color{red}{Q(s, a)} \right]$$</p>
  
  <p>In this update, the red term $\color{red}{Q(s, a)}$ (which appears twice) is our old estimate of the value $Q(s, a)$ of being in state $s$ and executing action $a$. The blue term $\color{blue}{r + \gamma Q(s', a')}$ is a different version of estimating the same quantity $Q(s, a)$. This second version is assumed to be slightly more accurate, because it is not "just" a prediction, but it's a combination of: </p>
  
  <ul>
  <li>something that we really observed: $r$, plus </li>
  <li>a prediction: $\gamma Q(s', a')$</li>
  </ul>
  
  <p>Here, the $r$ component is the immediate reward that we observed after executing $a$ in $s$, and then $Q(s', a')$ is everything we expect to still be collecting afterwards (i.e., after executing $a$ in $s$ and transitioning to $s'$).</p>
  
  <p>Now, suppose that $s'$ is a terminal state, what rewards do we still expect to be collecting in the future within that same episode? Since $s'$ is terminal, and the episode has ended, there can only be one correct answer; we expect to collect exactly $0$ rewards in the future.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-24T15:02:14.150Z
Answers.LastActivityDate: 2018-08-24T15:55:00.353Z
Answers.LastEditDate: 2018-08-24T15:55:00.353Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>I recently started learning about reinforcement learning. Currently, I am trying to implement the <a href="http://www.cse.unsw.edu.au/%7Ecs9417ml/RL1/algorithms.html" rel="nofollow noreferrer">SARSA algorithm</a>. However, I do not know how to deal with <span class="math-container">$Q(s', a')$</span>, when <span class="math-container">$s'$</span> is the terminal state. First, there is no action to choose from in this state. Second, this <span class="math-container">$Q$</span>-factor will never be updated either because the episode ends when <span class="math-container">$s'$</span> is reached. Should I initialize <span class="math-container">$Q(s', a')$</span> to something other than a random number? Or should I just ignore the <span class="math-container">$Q$</span>-factors and simply feed the reward <span class="math-container">$r$</span> into the update?</p>
  
Tags: <reinforcement-learning><implementation><sarsa>
Title: How should I handle action selection in the terminal state when implementing SARSA?
UserDisplayName: Hai Nguyen
ThreadText: # How should I handle action selection in the terminal state when implementing SARSA?
  
  (Asked by: username_0 on 2017-08-04)
  
  I recently started learning about reinforcement learning. Currently, I am trying to implement the [SARSA algorithm](http://www.cse.unsw.edu.au/%7Ecs9417ml/RL1/algorithms.html). However, I do not know how to deal with $Q(s', a')$, when $s'$ is the terminal state. First, there is no action to choose from in this state. Second, this $Q$-factor will never be updated either because the episode ends when $s'$ is reached. Should I initialize $Q(s', a')$ to something other than a random number? Or should I just ignore the $Q$-factors and simply feed the reward $r$ into the update?
  
  **Tags:** reinforcement-learning, implementation, sarsa
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2018-08-24. Score: 5 | Accepted answer
  
  The value $Q(s', ~\cdot~)$ should always be implemented to simply be equal to $0$ for any terminal state $s'$ (the dot instead of an action as second argument there indicates that what I just wrote should hold for **any action**, as long as $s'$ is terminal).
  
  It is easier to understand why this should be the case by dissecting what the different terms in the update rule mean:
  
  $$Q(s, a) \gets \color{red}{Q(s, a)} + \alpha \left[ \color{blue}{r + \gamma Q(s', a')} - \color{red}{Q(s, a)} \right]$$
  
  In this update, the red term $\color{red}{Q(s, a)}$ (which appears twice) is our old estimate of the value $Q(s, a)$ of being in state $s$ and executing action $a$. The blue term $\color{blue}{r + \gamma Q(s', a')}$ is a different version of estimating the same quantity $Q(s, a)$. This second version is assumed to be slightly more accurate, because it is not "just" a prediction, but it's a combination of:
  
  * something that we really observed: $r$, plus
  * a prediction: $\gamma Q(s', a')$
  
  Here, the $r$ component is the immediate reward that we observed after executing $a$ in $s$, and then $Q(s', a')$ is everything we expect to still be collecting afterwards (i.e., after executing $a$ in $s$ and transitioning to $s'$).
  
  Now, suppose that $s'$ is a terminal state, what rewards do we still expect to be collecting in the future within that same episode? Since $s'$ is terminal, and the episode has ended, there can only be one correct answer; we expect to collect exactly $0$ rewards in the future.
  
  
  ---
  
  ### Answer by username_1 on 2017-08-05. Score: 2
  
  From the description of the algorithm you linked to, it says to 'repeat until s is terminal'. So one would end the episode at that point and your intuition holds.
  
  Practically, if one was implementing a reward function where a specific reward is associated with the end of the episode such as "r(robot ran into a wall) = -100" then one can imagine that there is a terminal state just after this 'wall hit' state so the agent could see this reward. The episode would then be at a terminal state so would end.
  
  
  ---
  
  