<p>I've been working on implementing a path control algorithm for a robotic arm in a MuJoCo environment, aiming to reach a fixed point. To eliminate the potential instability issues of MuJoCo, I've greatly simplified the environment, ensuring the target point remains constant throughout the training—meaning the arm is trained to reach the same fixed point in a completely identical setting.</p>
<p>However, despite the environmental simplifications, the training results aren't as expected. The attached graphs illustrate the training progress, where the Actor's loss is decreasing steadily (as desired), but the Critic's loss shows an initial decrease.</p>
<p><a href="https://i.sstatic.net/Ezdpq.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Ezdpq.png" alt="enter image description here" /></a></p>
<p>During training, the observation space includes the coordinates of the robotic arm, the coordinates of the target, and their relative distance. I've employed the following reward function.</p>
<pre><code>     def get_reward(self, observation):
        distance_reward = - distance * 200  
        step_n = self.step_count
        movement_penalty = 2 * step_n  
        reward = distance_reward - - movement_penalty
        if self.check_done(observation): 
            reward += 500
        return reward.
</code></pre>
<p>The learning function part of the MADDPG is implemented as follows:</p>
<pre><code>    def learn(self):
        if not self.buffer.ready():
            return
        critic_states, actor_states, actions, rewards, critic_states_next, actor_states_next, done = self.buffer.sample_buffer()
        
        critic_states = torch.tensor(critic_states, dtype=torch.float).to(self.device)
        rewards = torch.tensor(rewards, dtype=torch.float).to(self.device)
        critic_states_next = torch.tensor(critic_states_next, dtype=torch.float).to(self.device)
        done = torch.tensor(done, dtype=torch.bool).to(self.device)
        actions_list = [torch.tensor(actions[idx], dtype=torch.float).to(self.device) for idx in range(self.n_agents)] 

         # Recompute actions for all agents to update the critic  
        current_actions = [ag.actor(torch.tensor(actor_states[idx], dtype=torch.float).to(self.device)) for idx, ag in enumerate(self.agents)]
        current_actions = torch.cat(current_actions, dim=1) 
        # Compute target actions for the next states  
        target_actions = [ag.target_actor(torch.tensor(actor_states_next[idx], dtype=torch.float).to(self.device)) for idx, ag in enumerate(self.agents)]
        target_actions = torch.cat(target_actions, dim=1)
        action_list = [actions_list[idx] for idx, ag in enumerate(self.agents)]
        action_list = torch.cat(action_list, dim=1)

        critic_losses = []
        actor_losses = []
        for agent_idx, agent in enumerate(self.agents):
            # Update critic
            q_targets_next = agent.target_critic(critic_states_next, target_actions).flatten() 
            q_targets_next[done[:, agent_idx]] = 0.0 
            q_targets = rewards[:, agent_idx] + agent.gamma * q_targets_next  
            q_expected = agent.critic(critic_states, action_list).flatten() 
            critic_loss = F.mse_loss(q_expected, q_targets) 
            #print(f&quot;Agent {agent_idx}, Critic Loss: {critic_loss.item()}&quot;)
            agent.critic.optimizer.zero_grad() 
            if agent_idx == 0:  
                critic_loss.backward(retain_graph=True) 
            else:
                critic_loss.backward()
            #for name, param in agent.critic.named_parameters():
            # Update actor
            actor_loss = agent.critic(critic_states, current_actions).flatten()
            actor_loss = -torch.mean(actor_loss) 
            #print(f&quot;Agent {agent_idx}, Actor Loss: {actor_loss.item()}&quot;)
            agent.actor.optimizer.zero_grad()
            if agent_idx == 0:
                actor_loss.backward(retain_graph=True) 
            else:
                actor_loss.backward()
            #for name, param in agent.actor.named_parameters():
                #print(f&quot;{name} weight: {param.data}&quot;)
            critic_losses.append(critic_loss.item())
            actor_losses.append(actor_loss.item())

        # Logging the average losses and rewards to TensorBoard
        avg_critic_loss = sum(critic_losses) / len(critic_losses)
        avg_actor_loss = sum(actor_losses) / len(actor_losses)
        avg_reward = rewards.mean().item()

        self.writer.add_scalar('Loss/Critic', avg_critic_loss, self.log_step)
        self.writer.add_scalar('Loss/Actor', avg_actor_loss, self.log_step)
        self.writer.add_scalar('Reward', avg_reward, self.log_step)

        self.log_step += 1

        for agent_idx, agent in enumerate(self.agents):
            agent.critic.optimizer.step() 
            agent.actor.optimizer.step()
            agent.critic.optimizer.zero_grad() 
        for idx, agent in enumerate(self.agents):
            agent.update_network_parameters()
</code></pre>
<p>Lastly, the training process is conducted with these parameters.
actor learning rate: 0.0003    critic learning rate: 0.0005
soft update parameter: 0.01   discount factor: 0.7</p>
<p>Thank you in advance for your assistance!</p>

CC BY-SA 4.0
2024-03-20T18:24:19.123Z
why do you need MADDPG? to me it doesnt seem like a multi agent environment
Alberto
CC BY-SA 4.0
2024-03-21T03:17:53.460Z
I am currently just conducting code testing in a simple environment. I plan to enhance the environment later to accomplish more complex tasks.
Weitao Kang
CC BY-SA 4.0
2024-03-21T10:20:53.117Z
@WeitaoKang you should test it in a simple multi-agent environment then.
David
CC BY-SA 4.0
2024-03-26T09:28:48.727Z
I also experienced a similar curve when using DDPG. After inspection, I found that throughout the entire training process, the Q value I obtained through the net continued to increase. Regardless of the situation of critical loss, the Q value always develops in the direction of increasing.
Weitao Kang
<reinforcement-learning>
Critic Loss with MADDPG for Fixed-Point Reach Task in MuJoCo
Weitao Kang
# Critic Loss with MADDPG for Fixed-Point Reach Task in MuJoCo

(Asked by: username_0 on 2024-03-20)

I've been working on implementing a path control algorithm for a robotic arm in a MuJoCo environment, aiming to reach a fixed point. To eliminate the potential instability issues of MuJoCo, I've greatly simplified the environment, ensuring the target point remains constant throughout the training—meaning the arm is trained to reach the same fixed point in a completely identical setting.
However, despite the environmental simplifications, the training results aren't as expected. The attached graphs illustrate the training progress, where the Actor's loss is decreasing steadily (as desired), but the Critic's loss shows an initial decrease.

During training, the observation space includes the coordinates of the robotic arm, the coordinates of the target, and their relative distance. I've employed the following reward function.
     def get_reward(self, observation):
        distance_reward = - distance * 200  
        step_n = self.step_count
        movement_penalty = 2 * step_n  
        reward = distance_reward - - movement_penalty
        if self.check_done(observation): 
            reward += 500
        return reward.

The learning function part of the MADDPG is implemented as follows:
    def learn(self):
        if not self.buffer.ready():
            return
        critic_states, actor_states, actions, rewards, critic_states_next, actor_states_next, done = self.buffer.sample_buffer()
        
        critic_states = torch.tensor(critic_states, dtype=torch.float).to(self.device)
        rewards = torch.tensor(rewards, dtype=torch.float).to(self.device)
        critic_states_next = torch.tensor(critic_states_next, dtype=torch.float).to(self.device)
        done = torch.tensor(done, dtype=torch.bool).to(self.device)
        actions_list = [torch.tensor(actions[idx], dtype=torch.float).to(self.device) for idx in range(self.n_agents)] 

         # Recompute actions for all agents to update the critic  
        current_actions = [ag.actor(torch.tensor(actor_states[idx], dtype=torch.float).to(self.device)) for idx, ag in enumerate(self.agents)]
        current_actions = torch.cat(current_actions, dim=1) 
        # Compute target actions for the next states  
        target_actions = [ag.target_actor(torch.tensor(actor_states_next[idx], dtype=torch.float).to(self.device)) for idx, ag in enumerate(self.agents)]
        target_actions = torch.cat(target_actions, dim=1)
        action_list = [actions_list[idx] for idx, ag in enumerate(self.agents)]
        action_list = torch.cat(action_list, dim=1)

        critic_losses = []
        actor_losses = []
        for agent_idx, agent in enumerate(self.agents):
            # Update critic
            q_targets_next = agent.target_critic(critic_states_next, target_actions).flatten() 
            q_targets_next[done[:, agent_idx]] = 0.0 
            q_targets = rewards[:, agent_idx] + agent.gamma * q_targets_next  
            q_expected = agent.critic(critic_states, action_list).flatten() 
            critic_loss = F.mse_loss(q_expected, q_targets) 
            #print(f"Agent {agent_idx}, Critic Loss: {critic_loss.item()}")
            agent.critic.optimizer.zero_grad() 
            if agent_idx == 0:  
                critic_loss.backward(retain_graph=True) 
            else:
                critic_loss.backward()
            #for name, param in agent.critic.named_parameters():
            # Update actor
            actor_loss = agent.critic(critic_states, current_actions).flatten()
            actor_loss = -torch.mean(actor_loss) 
            #print(f"Agent {agent_idx}, Actor Loss: {actor_loss.item()}")
            agent.actor.optimizer.zero_grad()
            if agent_idx == 0:
                actor_loss.backward(retain_graph=True) 
            else:
                actor_loss.backward()
            #for name, param in agent.actor.named_parameters():
                #print(f"{name} weight: {param.data}")
            critic_losses.append(critic_loss.item())
            actor_losses.append(actor_loss.item())

        # Logging the average losses and rewards to TensorBoard
        avg_critic_loss = sum(critic_losses) / len(critic_losses)
        avg_actor_loss = sum(actor_losses) / len(actor_losses)
        avg_reward = rewards.mean().item()

        self.writer.add_scalar('Loss/Critic', avg_critic_loss, self.log_step)
        self.writer.add_scalar('Loss/Actor', avg_actor_loss, self.log_step)
        self.writer.add_scalar('Reward', avg_reward, self.log_step)

        self.log_step += 1

        for agent_idx, agent in enumerate(self.agents):
            agent.critic.optimizer.step() 
            agent.actor.optimizer.step()
            agent.critic.optimizer.zero_grad() 
        for idx, agent in enumerate(self.agents):
            agent.update_network_parameters()

Lastly, the training process is conducted with these parameters.
actor learning rate: 0.0003    critic learning rate: 0.0005
soft update parameter: 0.01   discount factor: 0.7
Thank you in advance for your assistance!

**Tags:** reinforcement-learning

> **Comments:**
> **username_1**: why do you need MADDPG? to me it doesnt seem like a multi agent environment  
> **username_0**: I am currently just conducting code testing in a simple environment. I plan to enhance the environment later to accomplish more complex tasks.  
> **username_2**: @WeitaoKang you should test it in a simple multi-agent environment then.  
> **username_0**: I also experienced a similar curve when using DDPG. After inspection, I found that throughout the entire training process, the Q value I obtained through the net continued to increase. Regardless of the situation of critical loss, the Q value always develops in the direction of increasing.  

---

## 0 answers

