Body: <p>I'm building an agent to solve the Taxi environment. I've seen this problem solved with Q-Learning algorithms but my DQN consistently fails to learn anything. The environment has a discrete observation space, I one-hot encode the state before feeding it to the DQN. I also went ahead to implement Hindsight Experience Replay to help the learning process but the DQN still doesn't learn anything. What can I do to fix this?</p>
  <p>I've heard that DQN doesn't excel at environments that require planning to succeed, if that's the case, which algorithms would work well for this environment?</p>
  <p>EDIT</p>
  <p>When I posted this question, my DQN was learning from only 2 step transitions, since this environment can go on for several timesteps without any positive reward, I updated the agent to use transitions of 200 steps. Since I'm using Hindsight Experience Replay, my agent is sure to receive rewards within 200 timesteps even if it didn't meet the goal. I tried this and my agent still hasn't improved, it continually performs worse than the random agent baseline. I checked the contents of the buffer, I observed transitions that do lead to several rewards because their goals have been modified during HER and yet the DQN agent doesn't learn anything.</p>
  <p>Also, I'm using TensorFlow's tf_agents for my implementation. Here's a <a href="https://colab.research.google.com/drive/1pdWrDe7ZwzZ2bOWTR6KlEClxMq5X-Bhl?usp=sharing" rel="nofollow noreferrer">link to the code</a>. I repurposed <a href="https://www.tensorflow.org/agents/tutorials/1_dqn_tutorial" rel="nofollow noreferrer">this example</a>.</p>
  <p>I hope this helps</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-20T23:26:31.820Z
Comments.Text: Could you explain how you are encoding the space. There are ~500 discrete states - have you encoded all of them separately or combined state information in some different way?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-21T05:47:39.197Z
Comments.Text: @NeilSlater I used one-hot encoding.
Comments.UserDisplayName: Eunovo
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-22T09:48:06.050Z
Comments.Text: It is very likely that you have an implementation detail wrong. It is not clear from your description what that could be.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-22T20:01:43.680Z
Comments.Text: @NeilSlater I added some links. Also, I have changed the encoding method as you will see in the code, it didn't make any difference though. Now, I'm focusing on how I set goals. Maybe the goals are too difficult?
Comments.UserDisplayName: Eunovo
Tags: <reinforcement-learning><dqn>
Title: DQN fails to learn useful policy for the Taxi environment (Dietterich 200)
UserDisplayName: Eunovo
ThreadText: # DQN fails to learn useful policy for the Taxi environment (Dietterich 200)
  
  (Asked by: username_0 on 2020-11-20)
  
  I'm building an agent to solve the Taxi environment. I've seen this problem solved with Q-Learning algorithms but my DQN consistently fails to learn anything. The environment has a discrete observation space, I one-hot encode the state before feeding it to the DQN. I also went ahead to implement Hindsight Experience Replay to help the learning process but the DQN still doesn't learn anything. What can I do to fix this?
  
  I've heard that DQN doesn't excel at environments that require planning to succeed, if that's the case, which algorithms would work well for this environment?
  
  EDIT
  
  When I posted this question, my DQN was learning from only 2 step transitions, since this environment can go on for several timesteps without any positive reward, I updated the agent to use transitions of 200 steps. Since I'm using Hindsight Experience Replay, my agent is sure to receive rewards within 200 timesteps even if it didn't meet the goal. I tried this and my agent still hasn't improved, it continually performs worse than the random agent baseline. I checked the contents of the buffer, I observed transitions that do lead to several rewards because their goals have been modified during HER and yet the DQN agent doesn't learn anything.
  
  Also, I'm using TensorFlow's tf\_agents for my implementation. Here's a [link to the code](https://colab.research.google.com/drive/1pdWrDe7ZwzZ2bOWTR6KlEClxMq5X-Bhl?usp=sharing). I repurposed [this example](https://www.tensorflow.org/agents/tutorials/1_dqn_tutorial).
  
  I hope this helps
  
  **Tags:** reinforcement-learning, dqn
  
  > **Comments:**
  > **username_1**: Could you explain how you are encoding the space. There are ~500 discrete states - have you encoded all of them separately or combined state information in some different way?  
  > **username_0**: @username_1 I used one-hot encoding.  
  > **username_1**: It is very likely that you have an implementation detail wrong. It is not clear from your description what that could be.  
  > **username_0**: @username_1 I added some links. Also, I have changed the encoding method as you will see in the code, it didn't make any difference though. Now, I'm focusing on how I set goals. Maybe the goals are too difficult?  
  
  ---
  
  ## 0 answers
  
  