Answers.Body: <p>Terminal state is always the same in the sense that it represents the same thing, that the episode is over. They don’t need to be the exact same state; for instance you could have an <span class="math-container">$n$</span> by <span class="math-container">$n$</span> grid world where the top right and bottom left states are terminal as when you reach these your agent dies. These are both terminal but not the same state.</p>
  <p>For chess it would be any state that when reached the game ends (regardless of win/draw/lose). The difference between these terminal states is what reward you will receive for reaching it.</p>
  <p>Finally, normal states are non-terminal states, so there is no difference.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-10T10:00:32.147Z
Answers.Text: It may also be a good idea to describe what an _absorbing state_ is, although this was not part of the question, but it's related.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-08T16:43:31.483Z
Answers.LastActivityDate: 2021-06-08T16:43:31.483Z
Answers.UserDisplayName: David
Body: <p>In Sutton &amp; Barto's <a href="http://incompleteideas.net/book/the-book.html" rel="nofollow noreferrer"><em>Reinforcement Learning: An Introduction</em></a>, page 54, the authors define the <em>terminal state</em> as following:</p>
  <blockquote>
  <p>Each episode ends in a special state called the <em>terminal state</em></p>
  </blockquote>
  <p>But the authors also say:</p>
  <blockquote>
  <p>the episodes can all be considered to
  end in the same terminal state, with different rewards for the different outcomes. Tasks
  with episodes of this kind are called episodic tasks.</p>
  </blockquote>
  <p>I believe there is also a fundamental difference between a <em>terminal state</em>, <em>nonterminal states</em> and plain, normal states:</p>
  <blockquote>
  <p>In episodic tasks we sometimes need to distinguish the set of all nonterminal states, denoted S, from the set of all states plus the terminal state, denoted S+.</p>
  </blockquote>
  <p>In the first quote, it appears as if the <em>terminal state</em> is just a term to describe the final state of an episode, but, from the second quote, I understand that the <em>terminal state</em> is the same no matter the outcome of the episode. If we consider the game of chess, what would we consider as a <em>terminal state</em>? Would it be the state that, if reached, will end the game (checkmate), no matter the result (win, loss)? But then how can we describe a state that would lead to draw? If we say about a state that leads to a draw that it's a <em>nonterminal state</em> since we can play an &quot;infinite&quot; number of turns without reaching a win or a loss hence without reaching the <em>terminal state</em>, aren't we implicitly supposing that reaching a draw isn't a result for which we should attribute a reward (e.g. 0)? And if we name a state that leads to a draw a <em>terminal state</em>, then what would be the difference between a normal state and a <em>nonterminal state</em>?</p>
  
Tags: <reinforcement-learning><comparison><terminology><markov-decision-process><state-spaces>
Title: What is the difference between terminal state, nonterminal states and normal states?
UserDisplayName: Daviiid
ThreadText: # What is the difference between terminal state, nonterminal states and normal states?
  
  (Asked by: username_0 on 2021-06-08)
  
  In Sutton & Barto's [*Reinforcement Learning: An Introduction*](http://incompleteideas.net/book/the-book.html), page 54, the authors define the *terminal state* as following:
  
  > Each episode ends in a special state called the *terminal state*
  
  But the authors also say:
  
  > the episodes can all be considered to
  > end in the same terminal state, with different rewards for the different outcomes. Tasks
  > with episodes of this kind are called episodic tasks.
  
  I believe there is also a fundamental difference between a *terminal state*, *nonterminal states* and plain, normal states:
  
  > In episodic tasks we sometimes need to distinguish the set of all nonterminal states, denoted S, from the set of all states plus the terminal state, denoted S+.
  
  In the first quote, it appears as if the *terminal state* is just a term to describe the final state of an episode, but, from the second quote, I understand that the *terminal state* is the same no matter the outcome of the episode. If we consider the game of chess, what would we consider as a *terminal state*? Would it be the state that, if reached, will end the game (checkmate), no matter the result (win, loss)? But then how can we describe a state that would lead to draw? If we say about a state that leads to a draw that it's a *nonterminal state* since we can play an "infinite" number of turns without reaching a win or a loss hence without reaching the *terminal state*, aren't we implicitly supposing that reaching a draw isn't a result for which we should attribute a reward (e.g. 0)? And if we name a state that leads to a draw a *terminal state*, then what would be the difference between a normal state and a *nonterminal state*?
  
  **Tags:** reinforcement-learning, comparison, terminology, markov-decision-process, state-spaces
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-06-08. Score: 3 | Accepted answer
  
  Terminal state is always the same in the sense that it represents the same thing, that the episode is over. They don’t need to be the exact same state; for instance you could have an $n$ by $n$ grid world where the top right and bottom left states are terminal as when you reach these your agent dies. These are both terminal but not the same state.
  
  For chess it would be any state that when reached the game ends (regardless of win/draw/lose). The difference between these terminal states is what reward you will receive for reaching it.
  
  Finally, normal states are non-terminal states, so there is no difference.
  
  > **Comments:**
  > **username_2**: It may also be a good idea to describe what an _absorbing state_ is, although this was not part of the question, but it's related.  
  
  ---
  
  