Answers.Body: <p>In terms of meaningful decisions that an agent might make, then states <span class="math-container">$S_1$</span> and <span class="math-container">$S_2$</span> do seem redundant.</p>
  <p>However, if you remove those states and the actions from them, you also remove the time step tracking that goes along with them, and this fundamentally affects the MDP.</p>
  <p>The question gives some hints about what will be affected at the end to help you answer it. This is a non-episodic (continuing) MDP without a terminal state, so you cannot use discount factor <span class="math-container">$\gamma = 1$</span>, because the expected return will be infinite. Any valid approach for value functions in continuing MDPs - using a discount factor, finite horizon or average reward - will give you different results in the reduced MDP due to the difference in counting time steps.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-17T07:04:22.217Z
Answers.LastActivityDate: 2021-09-17T07:27:44.950Z
Answers.LastEditDate: 2021-09-17T07:27:44.950Z
Answers.UserDisplayName: Neil Slater
Body: <p>I came across this question about MDP.</p>
  <p>From the look of it, it seems the full MDP is reducible if the discarded state only have 1 way in and out but is it really so if we change the discounted factor? I think there is some tricky part regarding this problem...</p>
  <p><a href="https://i.sstatic.net/PFGuO.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/PFGuO.jpg" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-17T11:00:48.387Z
Comments.Text: What is the question? The question from the book?
Comments.UserDisplayName: MrPlanck
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-17T13:20:34.453Z
Comments.Text: Can you also clarify where you took this screenshot from?
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><markov-decision-process><value-functions>
Title: Discard irrelavant states from a MDP
UserDisplayName: Kronic
ThreadText: # Discard irrelavant states from a MDP
  
  (Asked by: username_0 on 2021-09-16)
  
  I came across this question about MDP.
  
  From the look of it, it seems the full MDP is reducible if the discarded state only have 1 way in and out but is it really so if we change the discounted factor? I think there is some tricky part regarding this problem...
  
  [![enter image description here](https://i.sstatic.net/PFGuO.jpg)](https://i.sstatic.net/PFGuO.jpg)
  
  **Tags:** reinforcement-learning, markov-decision-process, value-functions
  > **Comments:**
  > **username_1**: What is the question? The question from the book?  
  > **username_2**: Can you also clarify where you took this screenshot from?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2021-09-17. Score: 1
  
  In terms of meaningful decisions that an agent might make, then states $S\_1$ and $S\_2$ do seem redundant.
  
  However, if you remove those states and the actions from them, you also remove the time step tracking that goes along with them, and this fundamentally affects the MDP.
  
  The question gives some hints about what will be affected at the end to help you answer it. This is a non-episodic (continuing) MDP without a terminal state, so you cannot use discount factor $\gamma = 1$, because the expected return will be infinite. Any valid approach for value functions in continuing MDPs - using a discount factor, finite horizon or average reward - will give you different results in the reduced MDP due to the difference in counting time steps.
  
  
  ---
  
  