Answers.Body: <p>In an episodic problem, absorbing states are implemented to make the maths work similarly to continuing tasks. It allows one set of equations to cover two types of MDP (continuing and episodic). For example, the definition of state value might be</p>
  <p><span class="math-container">$$v(s) = \mathbb{E}[\sum_{k=0}^{\infty} \gamma^k R_{t + k + 1} | S_t = s]$$</span></p>
  <p>This sum is poorly defined if <span class="math-container">$t$</span> cannot continue to infinity. Absorbing states fix that without changing the nature of the MDP significantly. Adding absorbing states is a form of mathematical <em>generalisation</em> that makes two otherwise different maths system behave identically from a theoretical standpoint.</p>
  <p>So the short answer is: Yes, you should consider <em>all</em> terminal states to be absorbing if you are using that theoretical construct.</p>
  <p>You should bear in mind that absorbing states do not usually represent anything physical. The absorbing states are only used to allow two sets of equations to be merged for convenience in the literature. In practical projects, e.g. if you implement code for an agent, absorbing states are not really something you need to care about. Instead you will end processing states/actions/rewards at any terminal state and start a new episode as necessary.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-15T15:36:29.410Z
Answers.Text: The OP states "absorbing states are commonly used to represent goals", but is that really the case? Or is the absorbing state a state where the agent (conveniently) transitions to after having reached the goal? See, for example, [this answer](https://ai.stackexchange.com/a/22919/2444). It might be worth clarifying this.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-15T16:56:55.647Z
Answers.Text: @nbro: Yes I read the "used to represent goals" as "used for episode completion states associated with high rewards". I'm not sure if that slightly wooly shorthand needs correcting here, as it was clear to me what the OP was asking about (i.e. do we add absorbing states when termination reward is low as well as when it is high?)
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-15T13:33:17.780Z
Answers.LastActivityDate: 2022-05-15T14:49:13.920Z
Answers.LastEditDate: 2022-05-15T14:49:13.920Z
Answers.UserDisplayName: Neil Slater
Body: <p>My understanding is that, in Markov decision processes, absorbing state are states which can transition only to themselves and that these transitions generate rewards of 0. I know that absorbing states are commonly used to represent goals, so an agent might get a non-zero reward when <em>first entering</em> the absorbing state, but all subsequent transitions generate 0 reward (effectively ending the episode).</p>
  <p>My question is whether it is appropriate to also represent 'total failure' as an absorbing state. e.g. if the task is for Rabbit to return safely to his burrow (goal state which gives positive reward and ends the episode) while avoiding Fox (failure state which gives negative reward and ends the episode), would both burrow and fox-encounter be absorbing states? If not, how should fox-encounter be represented?</p>
  
Tags: <reinforcement-learning><markov-decision-process><state-spaces><transition-model><episodic-tasks>
Title: Is it appropriate to represent 'total failure' as an absorbing state?
UserDisplayName: K--
ThreadText: # Is it appropriate to represent 'total failure' as an absorbing state?
  
  (Asked by: username_0 on 2022-05-15)
  
  My understanding is that, in Markov decision processes, absorbing state are states which can transition only to themselves and that these transitions generate rewards of 0. I know that absorbing states are commonly used to represent goals, so an agent might get a non-zero reward when *first entering* the absorbing state, but all subsequent transitions generate 0 reward (effectively ending the episode).
  
  My question is whether it is appropriate to also represent 'total failure' as an absorbing state. e.g. if the task is for Rabbit to return safely to his burrow (goal state which gives positive reward and ends the episode) while avoiding Fox (failure state which gives negative reward and ends the episode), would both burrow and fox-encounter be absorbing states? If not, how should fox-encounter be represented?
  
  **Tags:** reinforcement-learning, markov-decision-process, state-spaces, transition-model, episodic-tasks
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-05-15. Score: 3
  
  In an episodic problem, absorbing states are implemented to make the maths work similarly to continuing tasks. It allows one set of equations to cover two types of MDP (continuing and episodic). For example, the definition of state value might be
  
  $$v(s) = \mathbb{E}[\sum\_{k=0}^{\infty} \gamma^k R\_{t + k + 1} | S\_t = s]$$
  
  This sum is poorly defined if $t$ cannot continue to infinity. Absorbing states fix that without changing the nature of the MDP significantly. Adding absorbing states is a form of mathematical *generalisation* that makes two otherwise different maths system behave identically from a theoretical standpoint.
  
  So the short answer is: Yes, you should consider *all* terminal states to be absorbing if you are using that theoretical construct.
  
  You should bear in mind that absorbing states do not usually represent anything physical. The absorbing states are only used to allow two sets of equations to be merged for convenience in the literature. In practical projects, e.g. if you implement code for an agent, absorbing states are not really something you need to care about. Instead you will end processing states/actions/rewards at any terminal state and start a new episode as necessary.
  
  > **Comments:**
  > **username_2**: The OP states "absorbing states are commonly used to represent goals", but is that really the case? Or is the absorbing state a state where the agent (conveniently) transitions to after having reached the goal? See, for example, [this answer](https://ai.stackexchange.com/a/22919/2444). It might be worth clarifying this.  
  > **username_1**: @username_2: Yes I read the "used to represent goals" as "used for episode completion states associated with high rewards". I'm not sure if that slightly wooly shorthand needs correcting here, as it was clear to me what the OP was asking about (i.e. do we add absorbing states when termination reward is low as well as when it is high?)  
  
  ---
  
  