Body: <p>I've been working on implementing a path control algorithm for a robotic arm in a MuJoCo environment, aiming to reach a fixed point. To eliminate the potential instability issues of MuJoCo, I've greatly simplified the environment, ensuring the target point remains constant throughout the training—meaning the arm is trained to reach the same fixed point in a completely identical setting.</p>
  <p>However, despite the environmental simplifications, the training results aren't as expected. The attached graphs illustrate the training progress, where the Actor's loss is decreasing steadily (as desired), but the Critic's loss shows an initial decrease.</p>
  <p><a href="https://i.sstatic.net/Ezdpq.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Ezdpq.png" alt="enter image description here" /></a></p>
  <p>During training, the observation space includes the coordinates of the robotic arm, the coordinates of the target, and their relative distance. I've employed the following reward function.</p>
  <pre><code>     def get_reward(self, observation):
          distance_reward = - distance * 200  
          step_n = self.step_count
          movement_penalty = 2 * step_n  
          reward = distance_reward - - movement_penalty
          if self.check_done(observation): 
              reward += 500
          return reward.
  </code></pre>
  <p>The learning function part of the MADDPG is implemented as follows:</p>
  <pre><code>    def learn(self):
          if not self.buffer.ready():
              return
          critic_states, actor_states, actions, rewards, critic_states_next, actor_states_next, done = self.buffer.sample_buffer()
          
          critic_states = torch.tensor(critic_states, dtype=torch.float).to(self.device)
          rewards = torch.tensor(rewards, dtype=torch.float).to(self.device)
          critic_states_next = torch.tensor(critic_states_next, dtype=torch.float).to(self.device)
          done = torch.tensor(done, dtype=torch.bool).to(self.device)
          actions_list = [torch.tensor(actions[idx], dtype=torch.float).to(self.device) for idx in range(self.n_agents)] 
  
           # Recompute actions for all agents to update the critic  
          current_actions = [ag.actor(torch.tensor(actor_states[idx], dtype=torch.float).to(self.device)) for idx, ag in enumerate(self.agents)]
          current_actions = torch.cat(current_actions, dim=1) 
          # Compute target actions for the next states  
          target_actions = [ag.target_actor(torch.tensor(actor_states_next[idx], dtype=torch.float).to(self.device)) for idx, ag in enumerate(self.agents)]
          target_actions = torch.cat(target_actions, dim=1)
          action_list = [actions_list[idx] for idx, ag in enumerate(self.agents)]
          action_list = torch.cat(action_list, dim=1)
  
          critic_losses = []
          actor_losses = []
          for agent_idx, agent in enumerate(self.agents):
              # Update critic
              q_targets_next = agent.target_critic(critic_states_next, target_actions).flatten() 
              q_targets_next[done[:, agent_idx]] = 0.0 
              q_targets = rewards[:, agent_idx] + agent.gamma * q_targets_next  
              q_expected = agent.critic(critic_states, action_list).flatten() 
              critic_loss = F.mse_loss(q_expected, q_targets) 
              #print(f&quot;Agent {agent_idx}, Critic Loss: {critic_loss.item()}&quot;)
              agent.critic.optimizer.zero_grad() 
              if agent_idx == 0:  
                  critic_loss.backward(retain_graph=True) 
              else:
                  critic_loss.backward()
              #for name, param in agent.critic.named_parameters():
              # Update actor
              actor_loss = agent.critic(critic_states, current_actions).flatten()
              actor_loss = -torch.mean(actor_loss) 
              #print(f&quot;Agent {agent_idx}, Actor Loss: {actor_loss.item()}&quot;)
              agent.actor.optimizer.zero_grad()
              if agent_idx == 0:
                  actor_loss.backward(retain_graph=True) 
              else:
                  actor_loss.backward()
              #for name, param in agent.actor.named_parameters():
                  #print(f&quot;{name} weight: {param.data}&quot;)
              critic_losses.append(critic_loss.item())
              actor_losses.append(actor_loss.item())
  
          # Logging the average losses and rewards to TensorBoard
          avg_critic_loss = sum(critic_losses) / len(critic_losses)
          avg_actor_loss = sum(actor_losses) / len(actor_losses)
          avg_reward = rewards.mean().item()
  
          self.writer.add_scalar('Loss/Critic', avg_critic_loss, self.log_step)
          self.writer.add_scalar('Loss/Actor', avg_actor_loss, self.log_step)
          self.writer.add_scalar('Reward', avg_reward, self.log_step)
  
          self.log_step += 1
  
          for agent_idx, agent in enumerate(self.agents):
              agent.critic.optimizer.step() 
              agent.actor.optimizer.step()
              agent.critic.optimizer.zero_grad() 
          for idx, agent in enumerate(self.agents):
              agent.update_network_parameters()
  </code></pre>
  <p>Lastly, the training process is conducted with these parameters.
  actor learning rate: 0.0003    critic learning rate: 0.0005
  soft update parameter: 0.01   discount factor: 0.7</p>
  <p>Thank you in advance for your assistance!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-20T18:24:19.123Z
Comments.Text: why do you need MADDPG? to me it doesnt seem like a multi agent environment
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-21T03:17:53.460Z
Comments.Text: I am currently just conducting code testing in a simple environment. I plan to enhance the environment later to accomplish more complex tasks.
Comments.UserDisplayName: Weitao Kang
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-21T10:20:53.117Z
Comments.Text: @WeitaoKang you should test it in a simple multi-agent environment then.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-26T09:28:48.727Z
Comments.Text: I also experienced a similar curve when using DDPG. After inspection, I found that throughout the entire training process, the Q value I obtained through the net continued to increase. Regardless of the situation of critical loss, the Q value always develops in the direction of increasing.
Comments.UserDisplayName: Weitao Kang
Tags: <reinforcement-learning>
Title: Critic Loss with MADDPG for Fixed-Point Reach Task in MuJoCo
UserDisplayName: Weitao Kang
ThreadText: # Critic Loss with MADDPG for Fixed-Point Reach Task in MuJoCo
  
  (Asked by: username_0 on 2024-03-20)
  
  I've been working on implementing a path control algorithm for a robotic arm in a MuJoCo environment, aiming to reach a fixed point. To eliminate the potential instability issues of MuJoCo, I've greatly simplified the environment, ensuring the target point remains constant throughout the training—meaning the arm is trained to reach the same fixed point in a completely identical setting.
  However, despite the environmental simplifications, the training results aren't as expected. The attached graphs illustrate the training progress, where the Actor's loss is decreasing steadily (as desired), but the Critic's loss shows an initial decrease.
  
  During training, the observation space includes the coordinates of the robotic arm, the coordinates of the target, and their relative distance. I've employed the following reward function.
       def get_reward(self, observation):
          distance_reward = - distance * 200  
          step_n = self.step_count
          movement_penalty = 2 * step_n  
          reward = distance_reward - - movement_penalty
          if self.check_done(observation): 
              reward += 500
          return reward.
  
  The learning function part of the MADDPG is implemented as follows:
      def learn(self):
          if not self.buffer.ready():
              return
          critic_states, actor_states, actions, rewards, critic_states_next, actor_states_next, done = self.buffer.sample_buffer()
          
          critic_states = torch.tensor(critic_states, dtype=torch.float).to(self.device)
          rewards = torch.tensor(rewards, dtype=torch.float).to(self.device)
          critic_states_next = torch.tensor(critic_states_next, dtype=torch.float).to(self.device)
          done = torch.tensor(done, dtype=torch.bool).to(self.device)
          actions_list = [torch.tensor(actions[idx], dtype=torch.float).to(self.device) for idx in range(self.n_agents)] 
  
           # Recompute actions for all agents to update the critic  
          current_actions = [ag.actor(torch.tensor(actor_states[idx], dtype=torch.float).to(self.device)) for idx, ag in enumerate(self.agents)]
          current_actions = torch.cat(current_actions, dim=1) 
          # Compute target actions for the next states  
          target_actions = [ag.target_actor(torch.tensor(actor_states_next[idx], dtype=torch.float).to(self.device)) for idx, ag in enumerate(self.agents)]
          target_actions = torch.cat(target_actions, dim=1)
          action_list = [actions_list[idx] for idx, ag in enumerate(self.agents)]
          action_list = torch.cat(action_list, dim=1)
  
          critic_losses = []
          actor_losses = []
          for agent_idx, agent in enumerate(self.agents):
              # Update critic
              q_targets_next = agent.target_critic(critic_states_next, target_actions).flatten() 
              q_targets_next[done[:, agent_idx]] = 0.0 
              q_targets = rewards[:, agent_idx] + agent.gamma * q_targets_next  
              q_expected = agent.critic(critic_states, action_list).flatten() 
              critic_loss = F.mse_loss(q_expected, q_targets) 
              #print(f"Agent {agent_idx}, Critic Loss: {critic_loss.item()}")
              agent.critic.optimizer.zero_grad() 
              if agent_idx == 0:  
                  critic_loss.backward(retain_graph=True) 
              else:
                  critic_loss.backward()
              #for name, param in agent.critic.named_parameters():
              # Update actor
              actor_loss = agent.critic(critic_states, current_actions).flatten()
              actor_loss = -torch.mean(actor_loss) 
              #print(f"Agent {agent_idx}, Actor Loss: {actor_loss.item()}")
              agent.actor.optimizer.zero_grad()
              if agent_idx == 0:
                  actor_loss.backward(retain_graph=True) 
              else:
                  actor_loss.backward()
              #for name, param in agent.actor.named_parameters():
                  #print(f"{name} weight: {param.data}")
              critic_losses.append(critic_loss.item())
              actor_losses.append(actor_loss.item())
  
          # Logging the average losses and rewards to TensorBoard
          avg_critic_loss = sum(critic_losses) / len(critic_losses)
          avg_actor_loss = sum(actor_losses) / len(actor_losses)
          avg_reward = rewards.mean().item()
  
          self.writer.add_scalar('Loss/Critic', avg_critic_loss, self.log_step)
          self.writer.add_scalar('Loss/Actor', avg_actor_loss, self.log_step)
          self.writer.add_scalar('Reward', avg_reward, self.log_step)
  
          self.log_step += 1
  
          for agent_idx, agent in enumerate(self.agents):
              agent.critic.optimizer.step() 
              agent.actor.optimizer.step()
              agent.critic.optimizer.zero_grad() 
          for idx, agent in enumerate(self.agents):
              agent.update_network_parameters()
  
  Lastly, the training process is conducted with these parameters.
  actor learning rate: 0.0003    critic learning rate: 0.0005
  soft update parameter: 0.01   discount factor: 0.7
  Thank you in advance for your assistance!
  
  **Tags:** reinforcement-learning
  > **Comments:**
  > **username_1**: why do you need MADDPG? to me it doesnt seem like a multi agent environment  
  > **username_0**: I am currently just conducting code testing in a simple environment. I plan to enhance the environment later to accomplish more complex tasks.  
  > **username_2**: @WeitaoKang you should test it in a simple multi-agent environment then.  
  > **username_0**: I also experienced a similar curve when using DDPG. After inspection, I found that throughout the entire training process, the Q value I obtained through the net continued to increase. Regardless of the situation of critical loss, the Q value always develops in the direction of increasing.  
  
  ---
  
  ## 0 answers
  
  