Answers.Body: <p>I don't know if there is a general definition of the terminal state based on the MDP transition probabilities.</p>
  <p>But remember that we define our MDP problem in a <span class="math-container">$\mathbb{S}$</span> set of all possible states and a <span class="math-container">$\mathbb{A}(s)$</span> representing the set of all possible actions for each state. Based on that, probably there aren't any possible actions for the terminal state, so the transition probability <span class="math-container">$p(s', r |s_T, a)$</span> can be undefined. Based on that, in episodic tasks we need to define a the set of all possible terminal states <span class="math-container">$\mathbb{S}^+$</span> and distinguish this with <span class="math-container">$\mathbb{S}$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-28T01:19:27.160Z
Answers.LastActivityDate: 2021-01-28T01:19:27.160Z
Answers.UserDisplayName: Alexandre Krul
Answers.Body: <p>As far as I remember, <em>terminal</em> state is a state from which agent cannot escape, i.e if the agent reached this state, he will never escape. In mathematical notation can be written as:
  <span class="math-container">$$
  p(s^{'}, r|s_T,a) = \delta_{s^{'}s_T} \delta_{rr_{S_T}}
  $$</span>
  Where <span class="math-container">$\delta_{ab}$</span> is a Kronecker symbol, and by <span class="math-container">$r_{S_T}$</span> I mean the reward collected by the agent sitting in the terminal state from now till the end of the episode.</p>
  <p>This state doesn't have to be unique. Imagine a Markov chain as a set of points, representing states, and arrows between the states with associated probabilities. Nothing prevents you from defining MDP where several nodes have only one outgoing arrow pointing to itself with the probability 1.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-28T06:46:51.367Z
Answers.LastActivityDate: 2021-01-28T12:18:07.863Z
Answers.LastEditDate: 2021-01-28T12:18:07.863Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Body: <p>I'm learning the basics of RL and I'm struggling to understand the notion of terminal state in MDPs.</p>
  <p>To ask my question straightforwardly: is there a natural way to define the terminal state from the MDP transition probabilities <span class="math-container">$p(s',r|s,a)$</span>? If I need to be more restrictive, assume a game setting, for example, chess.</p>
  <p>My first hypothesis would be to define the terminal state as the state <span class="math-container">$s_T$</span> such that <span class="math-container">$p(s',r|s_T,a) = p(s',r|s_T)$</span>, a state from which the transition is independent of the agent's actions. But that does not seem quite right. First, there is no particular reason why this state should be unique. Second, from this definition, it could also just be an intermittent state of &quot;lag&quot;.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-28T12:16:51.080Z
Comments.Text: Wait, are you asking how would the transition model be defined for a terminal state of an MDP? From the equations, that's what you seem to be trying to do, but that's somehow a different question than defining the "terminal state from the transition model" (which is your question in the body). So, I suggest that you edit this post to clarify what your question really is.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><markov-decision-process><state-spaces>
Title: Is there a natural way to define the terminal state from the MDP transition probabilities $p(s',r|s,a)$?
UserDisplayName: Undead
ThreadText: # Is there a natural way to define the terminal state from the MDP transition probabilities $p(s',r|s,a)$?
  
  (Asked by: username_0 on 2021-01-28)
  
  I'm learning the basics of RL and I'm struggling to understand the notion of terminal state in MDPs.
  
  To ask my question straightforwardly: is there a natural way to define the terminal state from the MDP transition probabilities $p(s',r|s,a)$? If I need to be more restrictive, assume a game setting, for example, chess.
  
  My first hypothesis would be to define the terminal state as the state $s\_T$ such that $p(s',r|s\_T,a) = p(s',r|s\_T)$, a state from which the transition is independent of the agent's actions. But that does not seem quite right. First, there is no particular reason why this state should be unique. Second, from this definition, it could also just be an intermittent state of "lag".
  
  **Tags:** reinforcement-learning, markov-decision-process, state-spaces
  
  > **Comments:**
  > **username_1**: Wait, are you asking how would the transition model be defined for a terminal state of an MDP? From the equations, that's what you seem to be trying to do, but that's somehow a different question than defining the "terminal state from the transition model" (which is your question in the body). So, I suggest that you edit this post to clarify what your question really is.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-01-28. Score: 1
  
  I don't know if there is a general definition of the terminal state based on the MDP transition probabilities.
  
  But remember that we define our MDP problem in a $\mathbb{S}$ set of all possible states and a $\mathbb{A}(s)$ representing the set of all possible actions for each state. Based on that, probably there aren't any possible actions for the terminal state, so the transition probability $p(s', r |s\_T, a)$ can be undefined. Based on that, in episodic tasks we need to define a the set of all possible terminal states $\mathbb{S}^+$ and distinguish this with $\mathbb{S}$.
  
  
  ---
  
  ### Answer by username_3 on 2021-01-28. Score: 1
  
  As far as I remember, *terminal* state is a state from which agent cannot escape, i.e if the agent reached this state, he will never escape. In mathematical notation can be written as:
  $$
  p(s^{'}, r|s\_T,a) = \delta\_{s^{'}s\_T} \delta\_{rr\_{S\_T}}
  $$
  Where $\delta\_{ab}$ is a Kronecker symbol, and by $r\_{S\_T}$ I mean the reward collected by the agent sitting in the terminal state from now till the end of the episode.
  
  This state doesn't have to be unique. Imagine a Markov chain as a set of points, representing states, and arrows between the states with associated probabilities. Nothing prevents you from defining MDP where several nodes have only one outgoing arrow pointing to itself with the probability 1.
  
  
  ---
  
  