Body: <p>I am diving into Multi-Agent Reinforcement Learning and after reading some literature, I would like to clarify some approaches because I am not quite sure. Now for the following two cases it is clear that:</p>
  <ul>
  <li><p><strong>independent learning:</strong> one distinct policy per agent with distinct rewards per agent of the same reward function.
  <span class="math-container">$$ 
  \pi_{\text{ag1}}: \text{obs}_{\text{ag1}} 
  \rightarrow \text{a}_{\text{ag1}} 
  \rightarrow \text{r}_{\text{ag1}}
  $$</span>
  <span class="math-container">$$ 
  \pi_{\text{ag2}}: \text{obs}_{\text{ag2}} 
  \rightarrow \text{a}_{\text{ag2}} 
  \rightarrow \text{r}_{\text{ag2}}
  $$</span></p>
  </li>
  <li><p><strong>joint learning:</strong> one (global) policy making decisions for all agents simultaneously with one global reward
  <span class="math-container">$$
  \pi_{\text{ag}}: [\text{obs}_{\text{ag1}},\text{obs}_{\text{ag2}},\text{obs}_{\text{ag3}}] \rightarrow  [\text{a}_{\text{ag1}},\text{a}_{\text{ag2}},\text{a}_{\text{ag3}}]  \rightarrow  \text{r}_{\text{glob}}=\text{r}_{\text{ag1}} + \text{r}_{\text{ag2}} + \text{r}_{\text{ag3}}
  $$</span></p>
  </li>
  </ul>
  <p>But using the <strong>CTDE</strong> aproach (centralized critic decentralized execution), I am not quite sure how it should be correctly applied. There are the following options for policy and reward that can be implemented and combined:</p>
  <ol>
  <li><strong>distinct policy:</strong> each agent has its own policy: <span class="math-container">$\pi_{\text{ag_i}}$</span></li>
  <li><strong>shared policy:</strong> each agent uses the same policy: <span class="math-container">$\pi_{\text{ag}}$</span></li>
  <li><strong>distinct reward</strong> values to each agent (by same reward function): <span class="math-container">$\text{a}_{\text{ag_i}}\rightarrow\text{r}_{\text{ag_i}}$</span></li>
  <li><strong>shared/global reward</strong> given to all agents: <span class="math-container">$\text{a}_{\text{ag_i}}\rightarrow\text{r}_{\text{glob}}=\text{sum}(\text{r}_{\text{ag_i}})$</span></li>
  </ol>
  <p>I further know that it is possible to assign a partial fraction <span class="math-container">$\alpha$</span> to the distinct reward in 3). i.e. <span class="math-container">$\text{r}_{\text{ag_i}} + \alpha\, \text{sum}(\text{r}_{\text{ag_j\i}})$</span> in order to enhance cooperation.</p>
  <p>But what is the &quot;correct way&quot; of learning a coordinated behaviour in MARL with heterogeneous agents, especially when applying CTDE?</p>
  <p>My goal is to apply MARL in a game, where the heterogeneous agents have to attack and destroy opponents. Currently I am using CTDE, where agents of the same type are using the same shared policy and they get distinct reward values (of the same reward function). I also tried a shared reward, but did not notice a significant change.  I update my policies by PPO, so I don't use an explicit MARL algorithm.
  I just want to make sure I am going the correct way. It's not the issue with implementation, but rather a question about the whole concept.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-06T10:28:51.027Z
Comments.Text: the reward structure does not influence the category... independent learning means "everybody is trained assuming other agents are part of the env", thus the independent... joint is just a "big puppetier" controlling everybody (goes back to single agent)... CTDE is half way, centralizing the reward signal, but decentralizing the policy (every agent has his own local observation).... consider that CTDE usually works best in continuous case scenario exploiting the DDPG structure, in discrete case they don't improve that much
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-08T02:03:08.727Z
Comments.Text: Hi. You can use mathjax on this site. I'd recommend you edit your post to do it.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><machine-learning><multi-agent-systems><multi-agent-rl>
Title: How to correctly train policies in multi-agent RL?
UserDisplayName: thsolyt
ThreadText: # How to correctly train policies in multi-agent RL?
  
  (Asked by: username_0 on 2024-02-06)
  
  I am diving into Multi-Agent Reinforcement Learning and after reading some literature, I would like to clarify some approaches because I am not quite sure. Now for the following two cases it is clear that:
  
  * **independent learning:** one distinct policy per agent with distinct rewards per agent of the same reward function.
    $$
    \pi\_{\text{ag1}}: \text{obs}\_{\text{ag1}}
    \rightarrow \text{a}\_{\text{ag1}}
    \rightarrow \text{r}\_{\text{ag1}}
    $$
    $$
    \pi\_{\text{ag2}}: \text{obs}\_{\text{ag2}}
    \rightarrow \text{a}\_{\text{ag2}}
    \rightarrow \text{r}\_{\text{ag2}}
    $$
  * **joint learning:** one (global) policy making decisions for all agents simultaneously with one global reward
    $$
    \pi\_{\text{ag}}: [\text{obs}\_{\text{ag1}},\text{obs}\_{\text{ag2}},\text{obs}\_{\text{ag3}}] \rightarrow [\text{a}\_{\text{ag1}},\text{a}\_{\text{ag2}},\text{a}\_{\text{ag3}}] \rightarrow \text{r}\_{\text{glob}}=\text{r}\_{\text{ag1}} + \text{r}\_{\text{ag2}} + \text{r}\_{\text{ag3}}
    $$
  
  But using the **CTDE** aproach (centralized critic decentralized execution), I am not quite sure how it should be correctly applied. There are the following options for policy and reward that can be implemented and combined:
  
  1. **distinct policy:** each agent has its own policy: $\pi\_{\text{ag\_i}}$
  2. **shared policy:** each agent uses the same policy: $\pi\_{\text{ag}}$
  3. **distinct reward** values to each agent (by same reward function): $\text{a}\_{\text{ag\_i}}\rightarrow\text{r}\_{\text{ag\_i}}$
  4. **shared/global reward** given to all agents: $\text{a}\_{\text{ag\_i}}\rightarrow\text{r}\_{\text{glob}}=\text{sum}(\text{r}\_{\text{ag\_i}})$
  
  I further know that it is possible to assign a partial fraction $\alpha$ to the distinct reward in 3). i.e. $\text{r}\_{\text{ag\_i}} + \alpha\, \text{sum}(\text{r}\_{\text{ag\_j\i}})$ in order to enhance cooperation.
  
  But what is the "correct way" of learning a coordinated behaviour in MARL with heterogeneous agents, especially when applying CTDE?
  
  My goal is to apply MARL in a game, where the heterogeneous agents have to attack and destroy opponents. Currently I am using CTDE, where agents of the same type are using the same shared policy and they get distinct reward values (of the same reward function). I also tried a shared reward, but did not notice a significant change. I update my policies by PPO, so I don't use an explicit MARL algorithm.
  I just want to make sure I am going the correct way. It's not the issue with implementation, but rather a question about the whole concept.
  
  **Tags:** reinforcement-learning, machine-learning, multi-agent-systems, multi-agent-rl
  > **Comments:**
  > **username_1**: the reward structure does not influence the category... independent learning means "everybody is trained assuming other agents are part of the env", thus the independent... joint is just a "big puppetier" controlling everybody (goes back to single agent)... CTDE is half way, centralizing the reward signal, but decentralizing the policy (every agent has his own local observation).... consider that CTDE usually works best in continuous case scenario exploiting the DDPG structure, in discrete case they don't improve that much  
  > **username_2**: Hi. You can use mathjax on this site. I'd recommend you edit your post to do it.  
  
  ---
  
  ## 0 answers
  
  