Answers.Body: <p>In toy problems like the Short Corridor task, you can choose the state representation to explore a key property, such as the ability of a particular method to solve it. Often this is done to extremes and heavily simplified.</p>
  <p>That is what is going on here. The state space that the agent is allowed to use is made highly degenerate with respect to the problem. This stands in for perhaps more complex partially observable systems, but in a way that is really clear to the reader. Also, it is still possible to derive analytically what the best policy should be, so methods can be examined as to how well they deal with the core issue (here, that state data is ambiguous).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-07T07:50:52.210Z
Answers.LastActivityDate: 2020-10-07T07:50:52.210Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>You can choose those states, but is the agent aware of the state it is in? From the text, it seems that the agent cannot distinguish between the three states. Its observation function is completely uninformative.</p>
  <p>This is why a <em>stochastic</em> policy is what is needed. This is common for POMDPs, whereas for regular MDPs we can always find a <em>deterministic</em> policy that is guaranteed to be optimal.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-07T02:51:50.713Z
Answers.LastActivityDate: 2020-10-07T02:51:50.713Z
Answers.UserDisplayName: Robby Goetschalckx
Body: <p>This is the Short Corridor problem taken from the Sutton &amp; Barto book. Here it's written:</p>
  <blockquote>
  <p>The problem is difficult because all the states appear identical under the function approximation</p>
  </blockquote>
  <p>But this doesn't make much sense as we can always choose states as 0,1,2 and corresponding feature vectors as</p>
  <p>x(S = 0,right) = [1  0  0  0  0  0]<br />
  x(S = 0 , left)  = [0  1  0  0  0  0]<br />
  x(S = 1,right) = [0  0  1  0  0  0]<br />
  x(S = 1 , left) = [0  0  0  1  0  0]<br />
  x(S = 2,right) = [0  0  0  0  1  0]<br />
  x(S = 2 , left) = [0  0  0  0  0  1]\</p>
  <p>So why is it written that all the states appear identical under the function approximation?</p>
  <p><a href="https://i.sstatic.net/lU6Nm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/lU6Nm.png" alt="enter image description here" /></a></p>
  
Tags: <reinforcement-learning><environment><function-approximation><sutton-barto>
Title: Why do all states appear identical under the function approximation in the Short Corridor task?
UserDisplayName: ZERO NULLS
ThreadText: # Why do all states appear identical under the function approximation in the Short Corridor task?
  
  (Asked by: username_0 on 2020-10-07)
  
  This is the Short Corridor problem taken from the Sutton & Barto book. Here it's written:
  
  > The problem is difficult because all the states appear identical under the function approximation
  
  But this doesn't make much sense as we can always choose states as 0,1,2 and corresponding feature vectors as
  
  x(S = 0,right) = [1 0 0 0 0 0]  
  x(S = 0 , left) = [0 1 0 0 0 0]  
  x(S = 1,right) = [0 0 1 0 0 0]  
  x(S = 1 , left) = [0 0 0 1 0 0]  
  x(S = 2,right) = [0 0 0 0 1 0]  
  x(S = 2 , left) = [0 0 0 0 0 1]\
  
  So why is it written that all the states appear identical under the function approximation?
  
  [![enter image description here](https://i.sstatic.net/lU6Nm.png)](https://i.sstatic.net/lU6Nm.png)
  
  **Tags:** reinforcement-learning, environment, function-approximation, sutton-barto
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-10-07. Score: 4 | Accepted answer
  
  You can choose those states, but is the agent aware of the state it is in? From the text, it seems that the agent cannot distinguish between the three states. Its observation function is completely uninformative.
  
  This is why a *stochastic* policy is what is needed. This is common for POMDPs, whereas for regular MDPs we can always find a *deterministic* policy that is guaranteed to be optimal.
  
  
  ---
  
  ### Answer by username_1 on 2020-10-07. Score: 1
  
  In toy problems like the Short Corridor task, you can choose the state representation to explore a key property, such as the ability of a particular method to solve it. Often this is done to extremes and heavily simplified.
  
  That is what is going on here. The state space that the agent is allowed to use is made highly degenerate with respect to the problem. This stands in for perhaps more complex partially observable systems, but in a way that is really clear to the reader. Also, it is still possible to derive analytically what the best policy should be, so methods can be examined as to how well they deal with the core issue (here, that state data is ambiguous).
  
  
  ---
  
  