<p>I had to change the actions selection function for this and tune some hyper-parameters. Here's what I did to make it converge:</p>
<ul>
<li>Sampled the noise from a standard normal distribution instead of sampling randomly.</li>
<li>Changed the polyak constant (tau) from 0.99 to 0.001 (I didn't have an idea of what it should be, so I had just set it randomly in the first try)</li>
<li>Changed the hidden layer sizes of the critic network to [64, 64].</li>
<li>Removed the ReLU activation after the second layer in the critic network. Earlier the layer were stacked as (Linear, ReLU, Linear, ReLU, Linear). I changed it to (Linear, ReLU, Linear, Linear).</li>
<li>Changed max buffer size to 1000000</li>
<li>Changed the size of the batch_size to be sampled from 128 to 64</li>
</ul>
<p>This is the plot that I get now after training it for 75 episodes :</p>
<p><a href="https://i.sstatic.net/q84yc.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/q84yc.png" alt="enter image description here" /></a></p>

CC BY-SA 4.0
2023-12-14T09:15:04.653Z
Did you use max_time_steps = 5000? Cause I believe that the standard task has 200 and as such you wouldn't find a solution.
HenDoNR
CC BY-SA 4.0
2020-08-10T09:17:48.393Z
2020-08-10T09:17:48.393Z
Vedant Shah
<p>I am trying to implement Deep Deterministic policy gradient algorithm by referring to the paper <em>Continuous Control using Deep Reinforcement Learning</em> on the MountainCarContinuous-v0 gym environment. I am using 2 hidden Linear layers of size 32 for both the actor and the critic networks with ReLU activations and a Tanh activation for the output layer of the actor network. However, for some reason, algorithm doesn't seem to converge for some reason. I tried tuning the hyperparameters to no success.</p>
<ul>
<li>Code</li>
</ul>
<pre><code>import copy
import random
from collections import deque, namedtuple

import matplotlib.pyplot as plt
import torch
import torch.nn as nn
import torch.optim as optim

&quot;&quot;&quot;
Hyperparameters:

actor_layer_sizes
critic_layer_sizes
max_buffer_size
polyak_constant
max_time_steps
max_episodes
actor_lr
critic_lr
GAMMA
update_after
batch_size
&quot;&quot;&quot;

device = torch.device(&quot;cpu&quot;)
dtype = torch.double

Transition = namedtuple(
    &quot;Transition&quot;, (&quot;state&quot;, &quot;action&quot;, &quot;reward&quot;, &quot;next_state&quot;, &quot;done&quot;)
)


class agent:
    def __init__(
        self,
        env,
        actor_layer_sizes=[32, 32],
        critic_layer_sizes=[32, 32],
        max_buffer_size=2500,
    ):
        self.env = env
        (
            self.actor,
            self.critic,
            self.target_actor,
            self.target_critic,
        ) = self.make_models(actor_layer_sizes, critic_layer_sizes)
        self.replay_buffer = deque(maxlen=max_buffer_size)
        self.max_buffer_size = max_buffer_size

    def make_models(self, actor_layer_sizes, critic_layer_sizes):
        actor = (
            nn.Sequential(
                nn.Linear(
                    self.env.observation_space.shape[0],
                    actor_layer_sizes[0],
                ),
                nn.ReLU(),
                nn.Linear(actor_layer_sizes[0], actor_layer_sizes[1]),
                nn.ReLU(),
                nn.Linear(
                    actor_layer_sizes[1], self.env.action_space.shape[0]
                ), nn.Tanh()
            )
            .to(device)
            .to(dtype)
        )

        critic = (
            nn.Sequential(
                nn.Linear(
                    self.env.observation_space.shape[0]
                    + self.env.action_space.shape[0],
                    critic_layer_sizes[0],
                ),
                nn.ReLU(),
                nn.Linear(critic_layer_sizes[0], critic_layer_sizes[1]),
                nn.ReLU(),
                nn.Linear(critic_layer_sizes[1], 1),
            )
            .to(device)
            .to(dtype)
        )

        target_actor = copy.deepcopy(actor)    # Create a target actor network

        target_critic = copy.deepcopy(critic)   # Create a target critic network

        return actor, critic, target_actor, target_critic

    def select_action(self, state, noise_factor):         # Selects an action in exploratory manner
      with torch.no_grad():
        noisy_action = self.actor(state) + noise_factor * torch.randn(size = self.env.action_space.shape, device=device, dtype=dtype)
        action = torch.clamp(noisy_action, self.env.action_space.low[0], self.env.action_space.high[0])

        return action

    def store_transition(self, state, action, reward, next_state, done):             # Stores the transition to the replay buffer with a default maximum capacity of 2500
        if len(self.replay_buffer) &lt; self.max_buffer_size:
            self.replay_buffer.append(
                Transition(state, action, reward, next_state, done)
            )
        else:
            self.replay_buffer.popleft()
            self.replay_buffer.append(
                Transition(state, action, reward, next_state, done)
            )

    def sample_batch(self, batch_size=128):                                            # Samples a random batch of transitions for training
      return Transition(
            *[torch.cat(i) for i in [*zip(*random.sample(self.replay_buffer, min(len(self.replay_buffer), batch_size)))]]
        )


    def train(
        self,
        GAMMA=0.99,
        actor_lr=0.001,
        critic_lr=0.001,
        polyak_constant=0.99,
        max_time_steps=5000,
        max_episodes=200,
        update_after=1,
        batch_size=128,
        noise_factor=0.2,
    ):
        
        self.train_rewards_list = []
        actor_optimizer = optim.Adam(self.actor.parameters(), lr=actor_lr)
        critic_optimizer = optim.Adam(
            self.critic.parameters(), lr=critic_lr
        )
        print(&quot;Starting Training:\n&quot;)
        for e in range(max_episodes):
            state = self.env.reset()
            state = torch.tensor(state, device=device, dtype=dtype).unsqueeze(0)
            episode_reward = 0
            for t in range(max_time_steps):
                #self.env.render()
                action = self.select_action(state, noise_factor)               
                next_state, reward, done, _ = self.env.step(action[0])         # Sample a transition
                episode_reward += reward

                next_state = torch.tensor(next_state, device=device, dtype=dtype).unsqueeze(0)
                reward = torch.tensor(
                    [reward], device=device, dtype=dtype
                ).unsqueeze(0)
                done = torch.tensor(
                    [done], device=device, dtype=dtype
                ).unsqueeze(0)

                self.store_transition(                               
                    state, action, reward, next_state, done
                )                # Store the transition in the replay buffer

                state = next_state
                
                sample_batch = self.sample_batch(128)

                with torch.no_grad():                 # Determine the target for the critic to train on
                  target = sample_batch.reward + (1 - sample_batch.done) * GAMMA * self.target_critic(torch.cat((sample_batch.next_state, self.target_actor(sample_batch.next_state)), dim=1))
                
                # Train the critic on the sampled batch
                critic_loss = nn.MSELoss()(
                    target,
                    self.critic(
                        torch.cat(
                            (sample_batch.state, sample_batch.action), dim=1
                        )
                    ),
                )

                critic_optimizer.zero_grad()
                critic_loss.backward()
                critic_optimizer.step()

                actor_loss = -1 * torch.mean(
                  self.critic(torch.cat((sample_batch.state, self.actor(sample_batch.state)), dim=1))
                  )

                #Train the actor  
                actor_optimizer.zero_grad()
                actor_loss.backward()
                actor_optimizer.step()
                

                #if (((t + 1) % update_after) == 0):
                for actor_param, target_actor_param in zip(self.actor.parameters(), self.target_actor.parameters()):
                  target_actor_param.data = polyak_constant * actor_param.data + (1 - polyak_constant) * target_actor_param.data
                  
                for critic_param, target_critic_param in zip(self.critic.parameters(), self.target_critic.parameters()):
                  target_critic_param.data = polyak_constant * critic_param.data + (1 - polyak_constant) * target_critic_param.data

                if done:
                    print(
                        &quot;Completed episode {}/{}&quot;.format(
                            e + 1, max_episodes
                        )
                    )
                    break

            self.train_rewards_list.append(episode_reward)

        self.env.close()
        print(self.train_rewards_list)

    def plot(self, plot_type):
        if (plot_type == &quot;train&quot;):
            plt.plot(self.train_rewards_list)
            plt.show()
        elif (plot_type == &quot;test&quot;):
            plt.plot(self.test_rewards_list)
            plt.show()
        else:
            print(&quot;\nInvalid plot type&quot;)
</code></pre>
<ul>
<li>Train code snippet</li>
</ul>
<pre><code>import gym

env = gym.make(&quot;MountainCarContinuous-v0&quot;)

myagent = agent(env)
myagent.train(max_episodes=150)
myagent.plot(&quot;train&quot;)
</code></pre>
<p>The figure below shows the plot for episode reward vs episode number:</p>
<p><a href="https://i.sstatic.net/Nipus.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Nipus.png" alt="enter image description here" /></a></p>

CC BY-SA 4.0
2020-08-09T18:43:00.570Z
have you tried tuning the reward function? I've never played with the continuous case but I recall for the discrete action case I had to modify the reward function as otherwise the feedback is too sparse.
David
CC BY-SA 4.0
2020-08-09T19:58:32.170Z
try modifying it so the reward is based on how far from the top of the hill the vehicle is.
David
CC BY-SA 4.0
2020-08-09T20:09:04.303Z
The reward function is defined in the gym environment itself right? I am not sure how to modify it. Besides I have seen several implementation which converge without any such modifications
Vedant Shah
CC BY-SA 4.0
2020-08-09T20:13:50.620Z
when you store the reward just have another line after that defining your own reward. for it to work without any modification you need a lot more compute power and memory because as I mentioned your rewards are sparse.
David
<reinforcement-learning><python><policy-gradients><ddpg><gym>
DDPG doesn't converge for MountainCarContinuous-v0 gym environment
Vedant Shah
# DDPG doesn't converge for MountainCarContinuous-v0 gym environment

(Asked by: username_0 on 2020-08-09)

I am trying to implement Deep Deterministic policy gradient algorithm by referring to the paper Continuous Control using Deep Reinforcement Learning on the MountainCarContinuous-v0 gym environment. I am using 2 hidden Linear layers of size 32 for both the actor and the critic networks with ReLU activations and a Tanh activation for the output layer of the actor network. However, for some reason, algorithm doesn't seem to converge for some reason. I tried tuning the hyperparameters to no success.

Code

import copy
import random
from collections import deque, namedtuple

import matplotlib.pyplot as plt
import torch
import torch.nn as nn
import torch.optim as optim

"""
Hyperparameters:

actor_layer_sizes
critic_layer_sizes
max_buffer_size
polyak_constant
max_time_steps
max_episodes
actor_lr
critic_lr
GAMMA
update_after
batch_size
"""

device = torch.device("cpu")
dtype = torch.double

Transition = namedtuple(
    "Transition", ("state", "action", "reward", "next_state", "done")
)


class agent:
    def __init__(
        self,
        env,
        actor_layer_sizes=[32, 32],
        critic_layer_sizes=[32, 32],
        max_buffer_size=2500,
    ):
        self.env = env
        (
            self.actor,
            self.critic,
            self.target_actor,
            self.target_critic,
        ) = self.make_models(actor_layer_sizes, critic_layer_sizes)
        self.replay_buffer = deque(maxlen=max_buffer_size)
        self.max_buffer_size = max_buffer_size

    def make_models(self, actor_layer_sizes, critic_layer_sizes):
        actor = (
            nn.Sequential(
                nn.Linear(
                    self.env.observation_space.shape[0],
                    actor_layer_sizes[0],
                ),
                nn.ReLU(),
                nn.Linear(actor_layer_sizes[0], actor_layer_sizes[1]),
                nn.ReLU(),
                nn.Linear(
                    actor_layer_sizes[1], self.env.action_space.shape[0]
                ), nn.Tanh()
            )
            .to(device)
            .to(dtype)
        )

        critic = (
            nn.Sequential(
                nn.Linear(
                    self.env.observation_space.shape[0]
                    + self.env.action_space.shape[0],
                    critic_layer_sizes[0],
                ),
                nn.ReLU(),
                nn.Linear(critic_layer_sizes[0], critic_layer_sizes[1]),
                nn.ReLU(),
                nn.Linear(critic_layer_sizes[1], 1),
            )
            .to(device)
            .to(dtype)
        )

        target_actor = copy.deepcopy(actor)    # Create a target actor network

        target_critic = copy.deepcopy(critic)   # Create a target critic network

        return actor, critic, target_actor, target_critic

    def select_action(self, state, noise_factor):         # Selects an action in exploratory manner
      with torch.no_grad():
        noisy_action = self.actor(state) + noise_factor * torch.randn(size = self.env.action_space.shape, device=device, dtype=dtype)
        action = torch.clamp(noisy_action, self.env.action_space.low[0], self.env.action_space.high[0])

        return action

    def store_transition(self, state, action, reward, next_state, done):             # Stores the transition to the replay buffer with a default maximum capacity of 2500
        if len(self.replay_buffer) < self.max_buffer_size:
            self.replay_buffer.append(
                Transition(state, action, reward, next_state, done)
            )
        else:
            self.replay_buffer.popleft()
            self.replay_buffer.append(
                Transition(state, action, reward, next_state, done)
            )

    def sample_batch(self, batch_size=128):                                            # Samples a random batch of transitions for training
      return Transition(
            *[torch.cat(i) for i in [*zip(*random.sample(self.replay_buffer, min(len(self.replay_buffer), batch_size)))]]
        )


    def train(
        self,
        GAMMA=0.99,
        actor_lr=0.001,
        critic_lr=0.001,
        polyak_constant=0.99,
        max_time_steps=5000,
        max_episodes=200,
        update_after=1,
        batch_size=128,
        noise_factor=0.2,
    ):
        
        self.train_rewards_list = []
        actor_optimizer = optim.Adam(self.actor.parameters(), lr=actor_lr)
        critic_optimizer = optim.Adam(
            self.critic.parameters(), lr=critic_lr
        )
        print("Starting Training:\n")
        for e in range(max_episodes):
            state = self.env.reset()
            state = torch.tensor(state, device=device, dtype=dtype).unsqueeze(0)
            episode_reward = 0
            for t in range(max_time_steps):
                #self.env.render()
                action = self.select_action(state, noise_factor)               
                next_state, reward, done, _ = self.env.step(action[0])         # Sample a transition
                episode_reward += reward

                next_state = torch.tensor(next_state, device=device, dtype=dtype).unsqueeze(0)
                reward = torch.tensor(
                    [reward], device=device, dtype=dtype
                ).unsqueeze(0)
                done = torch.tensor(
                    [done], device=device, dtype=dtype
                ).unsqueeze(0)

                self.store_transition(                               
                    state, action, reward, next_state, done
                )                # Store the transition in the replay buffer

                state = next_state
                
                sample_batch = self.sample_batch(128)

                with torch.no_grad():                 # Determine the target for the critic to train on
                  target = sample_batch.reward + (1 - sample_batch.done) * GAMMA * self.target_critic(torch.cat((sample_batch.next_state, self.target_actor(sample_batch.next_state)), dim=1))
                
                # Train the critic on the sampled batch
                critic_loss = nn.MSELoss()(
                    target,
                    self.critic(
                        torch.cat(
                            (sample_batch.state, sample_batch.action), dim=1
                        )
                    ),
                )

                critic_optimizer.zero_grad()
                critic_loss.backward()
                critic_optimizer.step()

                actor_loss = -1 * torch.mean(
                  self.critic(torch.cat((sample_batch.state, self.actor(sample_batch.state)), dim=1))
                  )

                #Train the actor  
                actor_optimizer.zero_grad()
                actor_loss.backward()
                actor_optimizer.step()
                

                #if (((t + 1) % update_after) == 0):
                for actor_param, target_actor_param in zip(self.actor.parameters(), self.target_actor.parameters()):
                  target_actor_param.data = polyak_constant * actor_param.data + (1 - polyak_constant) * target_actor_param.data
                  
                for critic_param, target_critic_param in zip(self.critic.parameters(), self.target_critic.parameters()):
                  target_critic_param.data = polyak_constant * critic_param.data + (1 - polyak_constant) * target_critic_param.data

                if done:
                    print(
                        "Completed episode {}/{}".format(
                            e + 1, max_episodes
                        )
                    )
                    break

            self.train_rewards_list.append(episode_reward)

        self.env.close()
        print(self.train_rewards_list)

    def plot(self, plot_type):
        if (plot_type == "train"):
            plt.plot(self.train_rewards_list)
            plt.show()
        elif (plot_type == "test"):
            plt.plot(self.test_rewards_list)
            plt.show()
        else:
            print("\nInvalid plot type")


Train code snippet

import gym

env = gym.make("MountainCarContinuous-v0")

myagent = agent(env)
myagent.train(max_episodes=150)
myagent.plot("train")

The figure below shows the plot for episode reward vs episode number:

**Tags:** reinforcement-learning, python, policy-gradients, ddpg, gym

> **Comments:**
> **username_1**: have you tried tuning the reward function? I've never played with the continuous case but I recall for the discrete action case I had to modify the reward function as otherwise the feedback is too sparse.  
> **username_1**: try modifying it so the reward is based on how far from the top of the hill the vehicle is.  
> **username_0**: The reward function is defined in the gym environment itself right? I am not sure how to modify it. Besides I have seen several implementation which converge without any such modifications  
> **username_1**: when you store the reward just have another line after that defining your own reward. for it to work without any modification you need a lot more compute power and memory because as I mentioned your rewards are sparse.  

---

## 1 answers

### Answer by username_0 on 2020-08-10. Score: 2 | Accepted answer

I had to change the actions selection function for this and tune some hyper-parameters. Here's what I did to make it converge:

* Sampled the noise from a standard normal distribution instead of sampling randomly.
* Changed the polyak constant (tau) from 0.99 to 0.001 (I didn't have an idea of what it should be, so I had just set it randomly in the first try)
* Changed the hidden layer sizes of the critic network to [64, 64].
* Removed the ReLU activation after the second layer in the critic network. Earlier the layer were stacked as (Linear, ReLU, Linear, ReLU, Linear). I changed it to (Linear, ReLU, Linear, Linear).
* Changed max buffer size to 1000000
* Changed the size of the batch\_size to be sampled from 128 to 64

This is the plot that I get now after training it for 75 episodes :

[![enter image description here](https://i.sstatic.net/q84yc.png)](https://i.sstatic.net/q84yc.png)

> **Comments:**
> **username_2**: Did you use max_time_steps = 5000? Cause I believe that the standard task has 200 and as such you wouldn't find a solution.  

---

