Answers.Body: <p>Yes, this can be done and is widely applied in recent literature on multi-agent RL, <em>at least with the collaborative setting where agents are optimizing a shared reward</em>. This is also known as <strong>parameter sharing</strong> between agents. Note that, even if the agents share the parameters in their policy networks, you can still get different behavior if the agents get different inputs, e.g., perceive different observations, or have different LSTM states.</p>
  <p>In terms of the loss function, no changes are required compared to the case where each agent has its own network with private parameters, as long as you make sure the network with shared parameters correctly processes the individual inputs of each agent.</p>
  <p>A concrete example is mentioned <a href="https://arxiv.org/pdf/1902.04043.pdf" rel="nofollow noreferrer">here</a> - see Appendix C. They also use the trick of appending a one-hot encoding of the agent index with the observation to enable learning of policies that differ based on agent identity. Or, if you prefer to see an implementation of the idea in code, check out how agents are defined in <a href="https://github.com/oxwhirl/pymarl" rel="nofollow noreferrer">this repository</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-13T12:17:31.207Z
Answers.Text: very helpful answer. I am however skeptical that "no change" is required. If I calc loss for each agent and backpropagate individually, the network is no longer the same after the first weight update (and so the second agent's backprop is now on a different set of weights). I think for there to be no change it would require that the agents take actions and update sequentially, but this would not be valid if they moved in parallel, right?
Answers.UserDisplayName: Josh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-13T14:25:30.857Z
Answers.Text: That's a good point, this is true at least for the case of cooperative agents that are optimizing a shared reward - this is also the setting covered by the resources I linked. I've clarified this point. I am not quite sure about the competitive setting where each agent has its own reward function - perhaps parameter sharing generally is not terribly appropriate in such a setting (why would one tell competitors what policy one is using?).
Answers.UserDisplayName: mikkola
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-13T21:11:09.520Z
Answers.Text: In my case the agents are "cooperative" in that they share a team reward, but they have individual rewards related to their individual movement that helps them learn the behavior of the more complex team reward swiftly. I believe in such a case I should be able to backprop all at once either by averaging or summing and can find that out empirically.
Answers.UserDisplayName: Josh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-17T19:39:56.007Z
Answers.Text: Adding for relevance of any others reading later, I honestly just averaged the losses and it worked fine. In fact it reduced some of the variance in learning as one might intuitively expect.
Answers.UserDisplayName: Josh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-13T07:47:30.330Z
Answers.LastActivityDate: 2022-09-13T14:24:04.437Z
Answers.LastEditDate: 2022-09-13T14:24:04.437Z
Answers.UserDisplayName: mikkola
Body: <p>Say I have a simple multi-agent reinforcement learning problem using vanilla policy gradient methods (i.e. REINFORCE) that is currently running with one network per agent. If I can say that each of my agents:</p>
  <ul>
  <li>are all of the same class</li>
  <li>have ~equivalent environmental contexts (on average)</li>
  <li>have no privileged state relative to other agents</li>
  <li>performs updates equally to all other agents</li>
  <li>DO use LSTMs (but store and reset memory states separately)</li>
  <li>DO receive rewards for individual actions based on their individual states</li>
  </ul>
  <p>...is it possible to use one network for all agents so as to minimize training time? And if so, how do I combine rewards and generate losses? For instance, if I calculate loss as the -logprob * reward (in the REINFORCE case), could I simply sum or average this over all agents and then backprop accordingly?</p>
  
Tags: <reinforcement-learning><deep-rl><reinforce><multi-agent-systems><multi-agent-rl>
Title: How can rewards and loss calculation be extended to multiple agents in a vanilla policy gradient RL setting?
UserDisplayName: Josh
ThreadText: # How can rewards and loss calculation be extended to multiple agents in a vanilla policy gradient RL setting?
  
  (Asked by: username_0 on 2022-09-12)
  
  Say I have a simple multi-agent reinforcement learning problem using vanilla policy gradient methods (i.e. REINFORCE) that is currently running with one network per agent. If I can say that each of my agents:
  
  * are all of the same class
  * have ~equivalent environmental contexts (on average)
  * have no privileged state relative to other agents
  * performs updates equally to all other agents
  * DO use LSTMs (but store and reset memory states separately)
  * DO receive rewards for individual actions based on their individual states
  
  ...is it possible to use one network for all agents so as to minimize training time? And if so, how do I combine rewards and generate losses? For instance, if I calculate loss as the -logprob \* reward (in the REINFORCE case), could I simply sum or average this over all agents and then backprop accordingly?
  
  **Tags:** reinforcement-learning, deep-rl, reinforce, multi-agent-systems, multi-agent-rl
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-09-13. Score: 2 | Accepted answer
  
  Yes, this can be done and is widely applied in recent literature on multi-agent RL, *at least with the collaborative setting where agents are optimizing a shared reward*. This is also known as **parameter sharing** between agents. Note that, even if the agents share the parameters in their policy networks, you can still get different behavior if the agents get different inputs, e.g., perceive different observations, or have different LSTM states.
  
  In terms of the loss function, no changes are required compared to the case where each agent has its own network with private parameters, as long as you make sure the network with shared parameters correctly processes the individual inputs of each agent.
  
  A concrete example is mentioned [here](https://arxiv.org/pdf/1902.04043.pdf) - see Appendix C. They also use the trick of appending a one-hot encoding of the agent index with the observation to enable learning of policies that differ based on agent identity. Or, if you prefer to see an implementation of the idea in code, check out how agents are defined in [this repository](https://github.com/oxwhirl/pymarl).
  
  > **Comments:**
  > **username_0**: very helpful answer. I am however skeptical that "no change" is required. If I calc loss for each agent and backpropagate individually, the network is no longer the same after the first weight update (and so the second agent's backprop is now on a different set of weights). I think for there to be no change it would require that the agents take actions and update sequentially, but this would not be valid if they moved in parallel, right?  
  > **username_1**: That's a good point, this is true at least for the case of cooperative agents that are optimizing a shared reward - this is also the setting covered by the resources I linked. I've clarified this point. I am not quite sure about the competitive setting where each agent has its own reward function - perhaps parameter sharing generally is not terribly appropriate in such a setting (why would one tell competitors what policy one is using?).  
  > **username_0**: In my case the agents are "cooperative" in that they share a team reward, but they have individual rewards related to their individual movement that helps them learn the behavior of the more complex team reward swiftly. I believe in such a case I should be able to backprop all at once either by averaging or summing and can find that out empirically.  
  > **username_0**: Adding for relevance of any others reading later, I honestly just averaged the losses and it worked fine. In fact it reduced some of the variance in learning as one might intuitively expect.  
  
  ---
  
  