Answers.Body: <p>This . . .</p>
  
  <blockquote>
    <p>because for the agent it should be the same task to go to a certain point, regardless of whether it's on its way to pick up or to drop-off</p>
  </blockquote>
  
  <p>. . . might seem logical/intuitive to a person understanding the task, but it is not mathematically correct. The agent cannot "merge" states because they involve the same behaviour. It <em>must</em> count differences in state as the combinations are presented. Critically, heading towards the passenger location or heading towards the goal location are not in any way similar to the agent, unless you manipulate the state to make them so*.</p>
  
  <p>Eventually the taxi will learn very similar navigation behaviour for picking up and dropping off a passenger. However, using a basic RL agent it learns these very much separately, and must re-learn the navigation rules independently for each combination of passenger and goal location.</p>
  
  <p>An agent that learned navigation within the environment, and then combined it into different tasks might be an example of hierarchical reinforcement learning, transfer learning, or curriculum learning. These are more sophisticated learning approaches, but it is quite interesting that even very basic RL problems can demonstrate a use for higher level abstractions. Most agents used on the taxi problem don't do this though, as 500 states is really very easy to "brute force" using the simplest algorithms.</p>
  
  <hr>
  
  <p>* You could modify the state representation to rationalise the task and make it have less states, similar to your suggestion. For instance, have one "target" location which could either be pickup or drop off, and a boolean "carrying passenger" state component. That would indeed reduce the number of states. However, that has involved you as the problem designer simplifying the problem to make it easier for the agent. Given that this is a toy problem designed as a benchmark to see how different agents perform, by doing that you subvert the purpose of the environment. If you were creating an agent to work on a harder real world problem though, it might be a very good idea to look for symmetries and ways to simplify state representation which would speed up learning.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-07T10:24:51.240Z
Answers.Text: Thanks for being informative. keep up the spirit
Answers.UserDisplayName: quintumnia
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-07T11:08:13.900Z
Answers.Text: Both answers are very informative. Accepted the first one for giving a short, specific answer but thank you for your abstraction of the underlying problem!
Answers.UserDisplayName: F.M.F.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-07T10:20:10.023Z
Answers.LastActivityDate: 2019-07-07T10:31:55.663Z
Answers.LastEditDate: 2019-07-07T10:31:55.663Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>This is more of a combinatorics than AI question but regradless, the full state information for the environment is:  </p>
  
  <p><span class="math-container">$(taxi \space position, passenger \space position, destination \space position)$</span></p>
  
  <p>There are 25 possible taxi positions, 5 passenger positions and 4 destination positions making it <span class="math-container">$25 \cdot 5 \cdot 4 = 500$</span>, so the paper is correct.  </p>
  
  <p>You are also correct but you divided 1 objective into 2 objectives and you have 2 separate policies, a pickup policy and a dropoff policy. So your state information would be for each policy:</p>
  
  <p><span class="math-container">$(taxi \space position, destination \space position)$</span>  </p>
  
  <p>There are 25 possible taxi positions and 4 possible destination positions making it <span class="math-container">$25 \cdot 4 = 100$</span>. You have 2 policies so you have <span class="math-container">$200$</span> states.</p>
  
  <p><strong>EDIT</strong></p>
  
  <p>Actually in the second case, I think you could get away with only 1 policy where you would simply change the destination position once you pick up the passenger so you dont need 2 separate policies and you would have only <span class="math-container">$100$</span> states</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-07T10:13:15.007Z
Answers.LastActivityDate: 2019-07-07T10:19:06.280Z
Answers.LastEditDate: 2019-07-07T10:19:06.280Z
Answers.UserDisplayName: Brale
Body: <p>Dietterich, who introduced the <a href="https://arxiv.org/pdf/cs/9905014.pdf" rel="nofollow noreferrer">taxi environment</a> (see p. 9), states the following: In total there “are 500 [distinct] possible states: 25 squares, 5 locations for the passenger (counting the four starting locations and the taxi), and 4 destinations” (Dietterich, 2000, p. 9).</p>
  
  <p>However, in my opinion there are only 25 (grid) * 4 (locations) * 2 (passenger in car) = 200 different states, because for the agent it should be the same task to go to a certain point, regardless of whether it's on its way to pick up or to drop-off. Only the action at the destination is different which would be stored binary (passenger in car or not)</p>
  
  <p>Why does Dietterich come up with 500 states? </p>
  
Tags: <reinforcement-learning><combinatorics>
Title: Number of states in taxi environment (Dietterich 2000)
UserDisplayName: F.M.F.
ThreadText: # Number of states in taxi environment (Dietterich 2000)
  
  (Asked by: username_0 on 2019-07-07)
  
  Dietterich, who introduced the [taxi environment](https://arxiv.org/pdf/cs/9905014.pdf) (see p. 9), states the following: In total there “are 500 [distinct] possible states: 25 squares, 5 locations for the passenger (counting the four starting locations and the taxi), and 4 destinations” (Dietterich, 2000, p. 9).
  
  However, in my opinion there are only 25 (grid) \* 4 (locations) \* 2 (passenger in car) = 200 different states, because for the agent it should be the same task to go to a certain point, regardless of whether it's on its way to pick up or to drop-off. Only the action at the destination is different which would be stored binary (passenger in car or not)
  
  Why does Dietterich come up with 500 states?
  
  **Tags:** reinforcement-learning, combinatorics
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2019-07-07. Score: 3 | Accepted answer
  
  This is more of a combinatorics than AI question but regradless, the full state information for the environment is:
  
  $(taxi \space position, passenger \space position, destination \space position)$
  
  There are 25 possible taxi positions, 5 passenger positions and 4 destination positions making it $25 \cdot 5 \cdot 4 = 500$, so the paper is correct.
  
  You are also correct but you divided 1 objective into 2 objectives and you have 2 separate policies, a pickup policy and a dropoff policy. So your state information would be for each policy:
  
  $(taxi \space position, destination \space position)$
  
  There are 25 possible taxi positions and 4 possible destination positions making it $25 \cdot 4 = 100$. You have 2 policies so you have $200$ states.
  
  **EDIT**
  
  Actually in the second case, I think you could get away with only 1 policy where you would simply change the destination position once you pick up the passenger so you dont need 2 separate policies and you would have only $100$ states
  
  
  ---
  
  ### Answer by username_1 on 2019-07-07. Score: 2
  
  This . . .
  
  > because for the agent it should be the same task to go to a certain point, regardless of whether it's on its way to pick up or to drop-off
  
  . . . might seem logical/intuitive to a person understanding the task, but it is not mathematically correct. The agent cannot "merge" states because they involve the same behaviour. It *must* count differences in state as the combinations are presented. Critically, heading towards the passenger location or heading towards the goal location are not in any way similar to the agent, unless you manipulate the state to make them so\*.
  
  Eventually the taxi will learn very similar navigation behaviour for picking up and dropping off a passenger. However, using a basic RL agent it learns these very much separately, and must re-learn the navigation rules independently for each combination of passenger and goal location.
  
  An agent that learned navigation within the environment, and then combined it into different tasks might be an example of hierarchical reinforcement learning, transfer learning, or curriculum learning. These are more sophisticated learning approaches, but it is quite interesting that even very basic RL problems can demonstrate a use for higher level abstractions. Most agents used on the taxi problem don't do this though, as 500 states is really very easy to "brute force" using the simplest algorithms.
  
  ---
  
  \* You could modify the state representation to rationalise the task and make it have less states, similar to your suggestion. For instance, have one "target" location which could either be pickup or drop off, and a boolean "carrying passenger" state component. That would indeed reduce the number of states. However, that has involved you as the problem designer simplifying the problem to make it easier for the agent. Given that this is a toy problem designed as a benchmark to see how different agents perform, by doing that you subvert the purpose of the environment. If you were creating an agent to work on a harder real world problem though, it might be a very good idea to look for symmetries and ways to simplify state representation which would speed up learning.
  
  > **Comments:**
  > **username_2**: Thanks for being informative. keep up the spirit  
  > **username_0**: Both answers are very informative. Accepted the first one for giving a short, specific answer but thank you for your abstraction of the underlying problem!  
  
  ---
  
  