Body: <p>I'm trying to solve the openai gym taxi problem (v3) using deep q learning.  I've already had some success with the q-table approach, but for the life of me cannot manage to train a NN to learn a reasonable action policy.  I'm doing the training using an AWS p3.2xlarge instance.</p>
  <p>My approach is fairly straightforward, I set up the environment and agent, then run the training loop.</p>
  <p>My code more or less looks like this:</p>
  <pre><code>import gym
  from taxi_agent import Agent
  
  env = gym.make('Taxi-v3').env
  optimizer = Adam(learning_rate=0.001)
  agent = Agent(env, optimizer)
  
  batch_size = 32
  num_of_episodes = 200
  timesteps_per_episode = 120
  </code></pre>
  <p>The agent was cobbled together from various examples online:</p>
  <pre><code>import numpy as np
  import random
  from IPython.display import clear_output
  from collections import deque
  from tensorflow.keras import Model, Sequential
  from tensorflow.keras.layers import Dense, Embedding, Reshape
  from tensorflow.keras.callbacks import ModelCheckpoint, TensorBoard
  
  
  class Agent:
      def __init__(self, environment, optimizer):
          
          # Initialize atributes
          self._state_size = environment.observation_space.n
          self._action_size = environment.action_space.n
          self._optimizer = optimizer
          
          self.expirience_replay = deque(maxlen=2000)
          
          # Initialize discount and exploration rate
          self.gamma = 0.6
          self.epsilon = 0.5
          
          # Build networks
          self.q_network = self._build_compile_model()
          self.target_network = self._build_compile_model()
          self.align_target_model()
  
          #: Set up some callbacks
          self.checkpoint_filepath = 'checkpoints/'
          model_checkpoint_callback = ModelCheckpoint(
              filepath=self.checkpoint_filepath,
              save_weights_only=True,
              save_freq='epoch'
          )
          # tensorboard_callback = TensorBoard('logs', update_freq=1)
          self.model_callbacks = [model_checkpoint_callback]
          
          self.history = []
  
      def store(self, state, action, reward, next_state, terminated):
          self.expirience_replay.append((state, action, reward, next_state, terminated))
      
      def _build_compile_model(self):
          model = Sequential()
          model.add(Embedding(self._state_size, 10, input_length=1))
          model.add(Reshape((10,)))
          model.add(Dense(48, activation='tanh'))
          model.add(Dense(24, activation='tanh'))
          model.add(Dense(self._action_size, activation='linear'))
          
          model.compile(loss='mse', optimizer=self._optimizer)
          return model
  
      def restore_weights(self):
          path = self.checkpoint_filepath
          print(f&quot;restoring model weights from {path}&quot;)
          self.q_network.load_weights(path)
  
      def align_target_model(self):
          self.target_network.set_weights(self.q_network.get_weights())
      
      def act(self, state, environment):
          if np.random.rand() &lt;= self.epsilon:
              return environment.action_space.sample()
          
          q_values = self.q_network.predict(state)
          return np.argmax(q_values[0])
  
      def retrain(self, batch_size, epochs=1):
          minibatch = random.sample(self.expirience_replay, batch_size)
          
          for state, action, reward, next_state, terminated in minibatch:
              
              target = self.q_network.predict(state)
              
              if terminated:
                  target[0][action] = reward
              else:
                  t = self.target_network.predict(next_state)
                  target[0][action] = reward + self.gamma * np.amax(t)
              
              history = self.q_network.fit(state, target, epochs=1, verbose=0, callbacks=self.model_callbacks)
              self.history.append(history.history)
  </code></pre>
  <p>The training loop uses the agent to act in the environment up to a number of <code>batch_size actions</code>.  Next, it retrains the model based on a random sample of the experience for every subsequent timestep.</p>
  <p>I have it set to print out feedback whenever the environment terminates (achieves the objective).  In practice this never happens.</p>
  <p>I've reloaded trained models from weights and trained for cumulative 24 hours without much success.  I've also tried silly things like updating the target network after N steps just so it learns something - no luck.</p>
  <p>If I try to use my trained model to solve an example env instance, it just wants to move south other than the random actions it set to do 50% of the time.</p>
  <p>It would be great it someone could give me some advice towards what to try next.  I can keep playing around with hyperparameters but I don't have the best intuition around where to optimize my efforts.</p>
  <pre><code>iterations = 0
  state = env.reset()
  env.render()
  while not terminated:
      state = np.reshape(state, [1, 1])
      action = agent.act(next_state, env)
      next_state, reward, terminated, info = env.step(action) 
      next_state = state
      if iterations % 10: env.render()
      iterations += 1
      if iterations &gt; 1000: break
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-15T10:23:45.440Z
Comments.Text: Could you please put your **specific** question in the title so that people immediately understand what your **specific** question is?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-15T12:40:35.050Z
Comments.Text: I added more detail to the title.
Comments.UserDisplayName: Sledge
Tags: <reinforcement-learning><open-ai><gym>
Title: Open AI Taxi - Agent fails to learn an effective policy
UserDisplayName: Sledge
ThreadText: # Open AI Taxi - Agent fails to learn an effective policy
  
  (Asked by: username_0 on 2021-04-14)
  
  I'm trying to solve the openai gym taxi problem (v3) using deep q learning. I've already had some success with the q-table approach, but for the life of me cannot manage to train a NN to learn a reasonable action policy. I'm doing the training using an AWS p3.2xlarge instance.
  
  My approach is fairly straightforward, I set up the environment and agent, then run the training loop.
  
  My code more or less looks like this:
  
  ```
  import gym
  from taxi_agent import Agent
  
  env = gym.make('Taxi-v3').env
  optimizer = Adam(learning_rate=0.001)
  agent = Agent(env, optimizer)
  
  batch_size = 32
  num_of_episodes = 200
  timesteps_per_episode = 120
  
  ```
  
  The agent was cobbled together from various examples online:
  
  ```
  import numpy as np
  import random
  from IPython.display import clear_output
  from collections import deque
  from tensorflow.keras import Model, Sequential
  from tensorflow.keras.layers import Dense, Embedding, Reshape
  from tensorflow.keras.callbacks import ModelCheckpoint, TensorBoard
  
  class Agent:
      def __init__(self, environment, optimizer):
          
          # Initialize atributes
          self._state_size = environment.observation_space.n
          self._action_size = environment.action_space.n
          self._optimizer = optimizer
          
          self.expirience_replay = deque(maxlen=2000)
          
          # Initialize discount and exploration rate
          self.gamma = 0.6
          self.epsilon = 0.5
          
          # Build networks
          self.q_network = self._build_compile_model()
          self.target_network = self._build_compile_model()
          self.align_target_model()
  
          #: Set up some callbacks
          self.checkpoint_filepath = 'checkpoints/'
          model_checkpoint_callback = ModelCheckpoint(
              filepath=self.checkpoint_filepath,
              save_weights_only=True,
              save_freq='epoch'
          )
          # tensorboard_callback = TensorBoard('logs', update_freq=1)
          self.model_callbacks = [model_checkpoint_callback]
          
          self.history = []
  
      def store(self, state, action, reward, next_state, terminated):
          self.expirience_replay.append((state, action, reward, next_state, terminated))
      
      def _build_compile_model(self):
          model = Sequential()
          model.add(Embedding(self._state_size, 10, input_length=1))
          model.add(Reshape((10,)))
          model.add(Dense(48, activation='tanh'))
          model.add(Dense(24, activation='tanh'))
          model.add(Dense(self._action_size, activation='linear'))
          
          model.compile(loss='mse', optimizer=self._optimizer)
          return model
  
      def restore_weights(self):
          path = self.checkpoint_filepath
          print(f"restoring model weights from {path}")
          self.q_network.load_weights(path)
  
      def align_target_model(self):
          self.target_network.set_weights(self.q_network.get_weights())
      
      def act(self, state, environment):
          if np.random.rand() <= self.epsilon:
              return environment.action_space.sample()
          
          q_values = self.q_network.predict(state)
          return np.argmax(q_values[0])
  
      def retrain(self, batch_size, epochs=1):
          minibatch = random.sample(self.expirience_replay, batch_size)
          
          for state, action, reward, next_state, terminated in minibatch:
              
              target = self.q_network.predict(state)
              
              if terminated:
                  target[0][action] = reward
              else:
                  t = self.target_network.predict(next_state)
                  target[0][action] = reward + self.gamma * np.amax(t)
              
              history = self.q_network.fit(state, target, epochs=1, verbose=0, callbacks=self.model_callbacks)
              self.history.append(history.history)
  
  ```
  
  The training loop uses the agent to act in the environment up to a number of `batch_size actions`. Next, it retrains the model based on a random sample of the experience for every subsequent timestep.
  
  I have it set to print out feedback whenever the environment terminates (achieves the objective). In practice this never happens.
  
  I've reloaded trained models from weights and trained for cumulative 24 hours without much success. I've also tried silly things like updating the target network after N steps just so it learns something - no luck.
  
  If I try to use my trained model to solve an example env instance, it just wants to move south other than the random actions it set to do 50% of the time.
  
  It would be great it someone could give me some advice towards what to try next. I can keep playing around with hyperparameters but I don't have the best intuition around where to optimize my efforts.
  
  ```
  iterations = 0
  state = env.reset()
  env.render()
  while not terminated:
      state = np.reshape(state, [1, 1])
      action = agent.act(next_state, env)
      next_state, reward, terminated, info = env.step(action) 
      next_state = state
      if iterations % 10: env.render()
      iterations += 1
      if iterations > 1000: break
  
  ```
  
  **Tags:** reinforcement-learning, open-ai, gym
  
  > **Comments:**
  > **username_1**: Could you please put your **specific** question in the title so that people immediately understand what your **specific** question is?  
  > **username_0**: I added more detail to the title.  
  
  ---
  
  ## 0 answers
  
  