Answers.Body: <p>I had to change the actions selection function for this and tune some hyper-parameters. Here's what I did to make it converge:</p>
  <ul>
  <li>Sampled the noise from a standard normal distribution instead of sampling randomly.</li>
  <li>Changed the polyak constant (tau) from 0.99 to 0.001 (I didn't have an idea of what it should be, so I had just set it randomly in the first try)</li>
  <li>Changed the hidden layer sizes of the critic network to [64, 64].</li>
  <li>Removed the ReLU activation after the second layer in the critic network. Earlier the layer were stacked as (Linear, ReLU, Linear, ReLU, Linear). I changed it to (Linear, ReLU, Linear, Linear).</li>
  <li>Changed max buffer size to 1000000</li>
  <li>Changed the size of the batch_size to be sampled from 128 to 64</li>
  </ul>
  <p>This is the plot that I get now after training it for 75 episodes :</p>
  <p><a href="https://i.sstatic.net/q84yc.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/q84yc.png" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-14T09:15:04.653Z
Answers.Text: Did you use max_time_steps = 5000? Cause I believe that the standard task has 200 and as such you wouldn't find a solution.
Answers.UserDisplayName: HenDoNR
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-10T09:17:48.393Z
Answers.LastActivityDate: 2020-08-10T09:17:48.393Z
Answers.UserDisplayName: Vedant Shah
Body: <p>I am trying to implement Deep Deterministic policy gradient algorithm by referring to the paper <em>Continuous Control using Deep Reinforcement Learning</em> on the MountainCarContinuous-v0 gym environment. I am using 2 hidden Linear layers of size 32 for both the actor and the critic networks with ReLU activations and a Tanh activation for the output layer of the actor network. However, for some reason, algorithm doesn't seem to converge for some reason. I tried tuning the hyperparameters to no success.</p>
  <ul>
  <li>Code</li>
  </ul>
  <pre><code>import copy
  import random
  from collections import deque, namedtuple
  
  import matplotlib.pyplot as plt
  import torch
  import torch.nn as nn
  import torch.optim as optim
  
  &quot;&quot;&quot;
  Hyperparameters:
  
  actor_layer_sizes
  critic_layer_sizes
  max_buffer_size
  polyak_constant
  max_time_steps
  max_episodes
  actor_lr
  critic_lr
  GAMMA
  update_after
  batch_size
  &quot;&quot;&quot;
  
  device = torch.device(&quot;cpu&quot;)
  dtype = torch.double
  
  Transition = namedtuple(
      &quot;Transition&quot;, (&quot;state&quot;, &quot;action&quot;, &quot;reward&quot;, &quot;next_state&quot;, &quot;done&quot;)
  )
  
  
  class agent:
      def __init__(
          self,
          env,
          actor_layer_sizes=[32, 32],
          critic_layer_sizes=[32, 32],
          max_buffer_size=2500,
      ):
          self.env = env
          (
              self.actor,
              self.critic,
              self.target_actor,
              self.target_critic,
          ) = self.make_models(actor_layer_sizes, critic_layer_sizes)
          self.replay_buffer = deque(maxlen=max_buffer_size)
          self.max_buffer_size = max_buffer_size
  
      def make_models(self, actor_layer_sizes, critic_layer_sizes):
          actor = (
              nn.Sequential(
                  nn.Linear(
                      self.env.observation_space.shape[0],
                      actor_layer_sizes[0],
                  ),
                  nn.ReLU(),
                  nn.Linear(actor_layer_sizes[0], actor_layer_sizes[1]),
                  nn.ReLU(),
                  nn.Linear(
                      actor_layer_sizes[1], self.env.action_space.shape[0]
                  ), nn.Tanh()
              )
              .to(device)
              .to(dtype)
          )
  
          critic = (
              nn.Sequential(
                  nn.Linear(
                      self.env.observation_space.shape[0]
                      + self.env.action_space.shape[0],
                      critic_layer_sizes[0],
                  ),
                  nn.ReLU(),
                  nn.Linear(critic_layer_sizes[0], critic_layer_sizes[1]),
                  nn.ReLU(),
                  nn.Linear(critic_layer_sizes[1], 1),
              )
              .to(device)
              .to(dtype)
          )
  
          target_actor = copy.deepcopy(actor)    # Create a target actor network
  
          target_critic = copy.deepcopy(critic)   # Create a target critic network
  
          return actor, critic, target_actor, target_critic
  
      def select_action(self, state, noise_factor):         # Selects an action in exploratory manner
        with torch.no_grad():
          noisy_action = self.actor(state) + noise_factor * torch.randn(size = self.env.action_space.shape, device=device, dtype=dtype)
          action = torch.clamp(noisy_action, self.env.action_space.low[0], self.env.action_space.high[0])
  
          return action
  
      def store_transition(self, state, action, reward, next_state, done):             # Stores the transition to the replay buffer with a default maximum capacity of 2500
          if len(self.replay_buffer) &lt; self.max_buffer_size:
              self.replay_buffer.append(
                  Transition(state, action, reward, next_state, done)
              )
          else:
              self.replay_buffer.popleft()
              self.replay_buffer.append(
                  Transition(state, action, reward, next_state, done)
              )
  
      def sample_batch(self, batch_size=128):                                            # Samples a random batch of transitions for training
        return Transition(
              *[torch.cat(i) for i in [*zip(*random.sample(self.replay_buffer, min(len(self.replay_buffer), batch_size)))]]
          )
  
  
      def train(
          self,
          GAMMA=0.99,
          actor_lr=0.001,
          critic_lr=0.001,
          polyak_constant=0.99,
          max_time_steps=5000,
          max_episodes=200,
          update_after=1,
          batch_size=128,
          noise_factor=0.2,
      ):
          
          self.train_rewards_list = []
          actor_optimizer = optim.Adam(self.actor.parameters(), lr=actor_lr)
          critic_optimizer = optim.Adam(
              self.critic.parameters(), lr=critic_lr
          )
          print(&quot;Starting Training:\n&quot;)
          for e in range(max_episodes):
              state = self.env.reset()
              state = torch.tensor(state, device=device, dtype=dtype).unsqueeze(0)
              episode_reward = 0
              for t in range(max_time_steps):
                  #self.env.render()
                  action = self.select_action(state, noise_factor)               
                  next_state, reward, done, _ = self.env.step(action[0])         # Sample a transition
                  episode_reward += reward
  
                  next_state = torch.tensor(next_state, device=device, dtype=dtype).unsqueeze(0)
                  reward = torch.tensor(
                      [reward], device=device, dtype=dtype
                  ).unsqueeze(0)
                  done = torch.tensor(
                      [done], device=device, dtype=dtype
                  ).unsqueeze(0)
  
                  self.store_transition(                               
                      state, action, reward, next_state, done
                  )                # Store the transition in the replay buffer
  
                  state = next_state
                  
                  sample_batch = self.sample_batch(128)
  
                  with torch.no_grad():                 # Determine the target for the critic to train on
                    target = sample_batch.reward + (1 - sample_batch.done) * GAMMA * self.target_critic(torch.cat((sample_batch.next_state, self.target_actor(sample_batch.next_state)), dim=1))
                  
                  # Train the critic on the sampled batch
                  critic_loss = nn.MSELoss()(
                      target,
                      self.critic(
                          torch.cat(
                              (sample_batch.state, sample_batch.action), dim=1
                          )
                      ),
                  )
  
                  critic_optimizer.zero_grad()
                  critic_loss.backward()
                  critic_optimizer.step()
  
                  actor_loss = -1 * torch.mean(
                    self.critic(torch.cat((sample_batch.state, self.actor(sample_batch.state)), dim=1))
                    )
  
                  #Train the actor  
                  actor_optimizer.zero_grad()
                  actor_loss.backward()
                  actor_optimizer.step()
                  
  
                  #if (((t + 1) % update_after) == 0):
                  for actor_param, target_actor_param in zip(self.actor.parameters(), self.target_actor.parameters()):
                    target_actor_param.data = polyak_constant * actor_param.data + (1 - polyak_constant) * target_actor_param.data
                    
                  for critic_param, target_critic_param in zip(self.critic.parameters(), self.target_critic.parameters()):
                    target_critic_param.data = polyak_constant * critic_param.data + (1 - polyak_constant) * target_critic_param.data
  
                  if done:
                      print(
                          &quot;Completed episode {}/{}&quot;.format(
                              e + 1, max_episodes
                          )
                      )
                      break
  
              self.train_rewards_list.append(episode_reward)
  
          self.env.close()
          print(self.train_rewards_list)
  
      def plot(self, plot_type):
          if (plot_type == &quot;train&quot;):
              plt.plot(self.train_rewards_list)
              plt.show()
          elif (plot_type == &quot;test&quot;):
              plt.plot(self.test_rewards_list)
              plt.show()
          else:
              print(&quot;\nInvalid plot type&quot;)
  </code></pre>
  <ul>
  <li>Train code snippet</li>
  </ul>
  <pre><code>import gym
  
  env = gym.make(&quot;MountainCarContinuous-v0&quot;)
  
  myagent = agent(env)
  myagent.train(max_episodes=150)
  myagent.plot(&quot;train&quot;)
  </code></pre>
  <p>The figure below shows the plot for episode reward vs episode number:</p>
  <p><a href="https://i.sstatic.net/Nipus.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Nipus.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-09T18:43:00.570Z
Comments.Text: have you tried tuning the reward function? I've never played with the continuous case but I recall for the discrete action case I had to modify the reward function as otherwise the feedback is too sparse.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-09T19:58:32.170Z
Comments.Text: try modifying it so the reward is based on how far from the top of the hill the vehicle is.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-09T20:09:04.303Z
Comments.Text: The reward function is defined in the gym environment itself right? I am not sure how to modify it. Besides I have seen several implementation which converge without any such modifications
Comments.UserDisplayName: Vedant Shah
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-09T20:13:50.620Z
Comments.Text: when you store the reward just have another line after that defining your own reward. for it to work without any modification you need a lot more compute power and memory because as I mentioned your rewards are sparse.
Comments.UserDisplayName: David
Tags: <reinforcement-learning><python><policy-gradients><ddpg><gym>
Title: DDPG doesn't converge for MountainCarContinuous-v0 gym environment
UserDisplayName: Vedant Shah
ThreadText: # DDPG doesn't converge for MountainCarContinuous-v0 gym environment
  
  (Asked by: username_0 on 2020-08-09)
  
  I am trying to implement Deep Deterministic policy gradient algorithm by referring to the paper Continuous Control using Deep Reinforcement Learning on the MountainCarContinuous-v0 gym environment. I am using 2 hidden Linear layers of size 32 for both the actor and the critic networks with ReLU activations and a Tanh activation for the output layer of the actor network. However, for some reason, algorithm doesn't seem to converge for some reason. I tried tuning the hyperparameters to no success.
  
  Code
  
  import copy
  import random
  from collections import deque, namedtuple
  
  import matplotlib.pyplot as plt
  import torch
  import torch.nn as nn
  import torch.optim as optim
  
  """
  Hyperparameters:
  
  actor_layer_sizes
  critic_layer_sizes
  max_buffer_size
  polyak_constant
  max_time_steps
  max_episodes
  actor_lr
  critic_lr
  GAMMA
  update_after
  batch_size
  """
  
  device = torch.device("cpu")
  dtype = torch.double
  
  Transition = namedtuple(
      "Transition", ("state", "action", "reward", "next_state", "done")
  )
  
  
  class agent:
      def __init__(
          self,
          env,
          actor_layer_sizes=[32, 32],
          critic_layer_sizes=[32, 32],
          max_buffer_size=2500,
      ):
          self.env = env
          (
              self.actor,
              self.critic,
              self.target_actor,
              self.target_critic,
          ) = self.make_models(actor_layer_sizes, critic_layer_sizes)
          self.replay_buffer = deque(maxlen=max_buffer_size)
          self.max_buffer_size = max_buffer_size
  
      def make_models(self, actor_layer_sizes, critic_layer_sizes):
          actor = (
              nn.Sequential(
                  nn.Linear(
                      self.env.observation_space.shape[0],
                      actor_layer_sizes[0],
                  ),
                  nn.ReLU(),
                  nn.Linear(actor_layer_sizes[0], actor_layer_sizes[1]),
                  nn.ReLU(),
                  nn.Linear(
                      actor_layer_sizes[1], self.env.action_space.shape[0]
                  ), nn.Tanh()
              )
              .to(device)
              .to(dtype)
          )
  
          critic = (
              nn.Sequential(
                  nn.Linear(
                      self.env.observation_space.shape[0]
                      + self.env.action_space.shape[0],
                      critic_layer_sizes[0],
                  ),
                  nn.ReLU(),
                  nn.Linear(critic_layer_sizes[0], critic_layer_sizes[1]),
                  nn.ReLU(),
                  nn.Linear(critic_layer_sizes[1], 1),
              )
              .to(device)
              .to(dtype)
          )
  
          target_actor = copy.deepcopy(actor)    # Create a target actor network
  
          target_critic = copy.deepcopy(critic)   # Create a target critic network
  
          return actor, critic, target_actor, target_critic
  
      def select_action(self, state, noise_factor):         # Selects an action in exploratory manner
        with torch.no_grad():
          noisy_action = self.actor(state) + noise_factor * torch.randn(size = self.env.action_space.shape, device=device, dtype=dtype)
          action = torch.clamp(noisy_action, self.env.action_space.low[0], self.env.action_space.high[0])
  
          return action
  
      def store_transition(self, state, action, reward, next_state, done):             # Stores the transition to the replay buffer with a default maximum capacity of 2500
          if len(self.replay_buffer) < self.max_buffer_size:
              self.replay_buffer.append(
                  Transition(state, action, reward, next_state, done)
              )
          else:
              self.replay_buffer.popleft()
              self.replay_buffer.append(
                  Transition(state, action, reward, next_state, done)
              )
  
      def sample_batch(self, batch_size=128):                                            # Samples a random batch of transitions for training
        return Transition(
              *[torch.cat(i) for i in [*zip(*random.sample(self.replay_buffer, min(len(self.replay_buffer), batch_size)))]]
          )
  
  
      def train(
          self,
          GAMMA=0.99,
          actor_lr=0.001,
          critic_lr=0.001,
          polyak_constant=0.99,
          max_time_steps=5000,
          max_episodes=200,
          update_after=1,
          batch_size=128,
          noise_factor=0.2,
      ):
          
          self.train_rewards_list = []
          actor_optimizer = optim.Adam(self.actor.parameters(), lr=actor_lr)
          critic_optimizer = optim.Adam(
              self.critic.parameters(), lr=critic_lr
          )
          print("Starting Training:\n")
          for e in range(max_episodes):
              state = self.env.reset()
              state = torch.tensor(state, device=device, dtype=dtype).unsqueeze(0)
              episode_reward = 0
              for t in range(max_time_steps):
                  #self.env.render()
                  action = self.select_action(state, noise_factor)               
                  next_state, reward, done, _ = self.env.step(action[0])         # Sample a transition
                  episode_reward += reward
  
                  next_state = torch.tensor(next_state, device=device, dtype=dtype).unsqueeze(0)
                  reward = torch.tensor(
                      [reward], device=device, dtype=dtype
                  ).unsqueeze(0)
                  done = torch.tensor(
                      [done], device=device, dtype=dtype
                  ).unsqueeze(0)
  
                  self.store_transition(                               
                      state, action, reward, next_state, done
                  )                # Store the transition in the replay buffer
  
                  state = next_state
                  
                  sample_batch = self.sample_batch(128)
  
                  with torch.no_grad():                 # Determine the target for the critic to train on
                    target = sample_batch.reward + (1 - sample_batch.done) * GAMMA * self.target_critic(torch.cat((sample_batch.next_state, self.target_actor(sample_batch.next_state)), dim=1))
                  
                  # Train the critic on the sampled batch
                  critic_loss = nn.MSELoss()(
                      target,
                      self.critic(
                          torch.cat(
                              (sample_batch.state, sample_batch.action), dim=1
                          )
                      ),
                  )
  
                  critic_optimizer.zero_grad()
                  critic_loss.backward()
                  critic_optimizer.step()
  
                  actor_loss = -1 * torch.mean(
                    self.critic(torch.cat((sample_batch.state, self.actor(sample_batch.state)), dim=1))
                    )
  
                  #Train the actor  
                  actor_optimizer.zero_grad()
                  actor_loss.backward()
                  actor_optimizer.step()
                  
  
                  #if (((t + 1) % update_after) == 0):
                  for actor_param, target_actor_param in zip(self.actor.parameters(), self.target_actor.parameters()):
                    target_actor_param.data = polyak_constant * actor_param.data + (1 - polyak_constant) * target_actor_param.data
                    
                  for critic_param, target_critic_param in zip(self.critic.parameters(), self.target_critic.parameters()):
                    target_critic_param.data = polyak_constant * critic_param.data + (1 - polyak_constant) * target_critic_param.data
  
                  if done:
                      print(
                          "Completed episode {}/{}".format(
                              e + 1, max_episodes
                          )
                      )
                      break
  
              self.train_rewards_list.append(episode_reward)
  
          self.env.close()
          print(self.train_rewards_list)
  
      def plot(self, plot_type):
          if (plot_type == "train"):
              plt.plot(self.train_rewards_list)
              plt.show()
          elif (plot_type == "test"):
              plt.plot(self.test_rewards_list)
              plt.show()
          else:
              print("\nInvalid plot type")
  
  
  Train code snippet
  
  import gym
  
  env = gym.make("MountainCarContinuous-v0")
  
  myagent = agent(env)
  myagent.train(max_episodes=150)
  myagent.plot("train")
  
  The figure below shows the plot for episode reward vs episode number:
  
  **Tags:** reinforcement-learning, python, policy-gradients, ddpg, gym
  > **Comments:**
  > **username_1**: have you tried tuning the reward function? I've never played with the continuous case but I recall for the discrete action case I had to modify the reward function as otherwise the feedback is too sparse.  
  > **username_1**: try modifying it so the reward is based on how far from the top of the hill the vehicle is.  
  > **username_0**: The reward function is defined in the gym environment itself right? I am not sure how to modify it. Besides I have seen several implementation which converge without any such modifications  
  > **username_1**: when you store the reward just have another line after that defining your own reward. for it to work without any modification you need a lot more compute power and memory because as I mentioned your rewards are sparse.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-08-10. Score: 2 | Accepted answer
  
  I had to change the actions selection function for this and tune some hyper-parameters. Here's what I did to make it converge:
  
  * Sampled the noise from a standard normal distribution instead of sampling randomly.
  * Changed the polyak constant (tau) from 0.99 to 0.001 (I didn't have an idea of what it should be, so I had just set it randomly in the first try)
  * Changed the hidden layer sizes of the critic network to [64, 64].
  * Removed the ReLU activation after the second layer in the critic network. Earlier the layer were stacked as (Linear, ReLU, Linear, ReLU, Linear). I changed it to (Linear, ReLU, Linear, Linear).
  * Changed max buffer size to 1000000
  * Changed the size of the batch\_size to be sampled from 128 to 64
  
  This is the plot that I get now after training it for 75 episodes :
  
  [![enter image description here](https://i.sstatic.net/q84yc.png)](https://i.sstatic.net/q84yc.png)
  
  > **Comments:**
  > **username_2**: Did you use max_time_steps = 5000? Cause I believe that the standard task has 200 and as such you wouldn't find a solution.  
  
  ---
  
  