Body: <p>I am experimenting with MADDPG algorithm implemented in this <a href="https://github.com/xuehy/pytorch-maddpg/tree/b7c1acf027a64a492eee6ab1c304ed3b8e69b5c4" rel="nofollow noreferrer">repo</a>. Since there were only a few agents (2-3) in the implementation (also in the original paper) steps like parameter updates, action prediction, etc. are done in a for loop. I want to increase the number of agents, say 10 or 30, and perform parallelization of the above-mentioned steps for all agents, i.e. I want to avoid for loops like this</p>
  <pre><code>for agent_idx in range(n_agents):
      ...
      ...
  </code></pre>
  <p>I tried Python Multiprocessing module with pool.map method but I am getting 'AttributeError: Can't pickle local object ...&quot;. Below is code I am running to get a joint action prediction but it results in the error above.</p>
  <pre><code>def get_ind_action(i, obs_i):
      return actor_critic[i].act(obs_i) # returns an individual action for a given observation for ith agent
  
  def get_joint_action(obs):
      pool = Pool()
      args_list = [[i, obs[i]] for i in range(n_agents)]
      joint_action = pool.map(get_ind_action, args_list)
      return joint_action
      
  </code></pre>
  <p>Here actor_critic is a list of neural networks of all agents, obs is the joint state observed by the centralized critic but each actor only sees its own state. The algorithm has the following architecture.</p>
  <p><a href="https://i.sstatic.net/PH8n3.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/PH8n3.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-06T06:34:38.327Z
Comments.Text: Hey, maybe to help someone unfamiliar with MADDPG (like me) to quickly grasp the difficulty of parallelizing it can you clarify that (a) are the agents gonna share info about their parameters at some point? (b) and if any other information is going to be shared among process in the learning phase? (c) are you perhaps planning to parallelize on single node or multi nodes? (d) to answer the error perhaps share short reproducible parts of the code that you add to the code you linked
Comments.UserDisplayName: Sanyou
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-08T00:44:28.990Z
Comments.Text: (a) they have identical architecture, i.e. the same number of layers and units in each layer but share no parameters. (b) no information is shared among agents' actor networks. (c) single node. (d) added above
Comments.UserDisplayName: Mika
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-08T01:27:38.130Z
Comments.Text: You have to pass the agent as the argument to the function, because the subprocess do not have the agent in the memory. You might want to pass the actor's state_dict instead of the agent and reinstantiate the agent in the subprocess->load the state_dict -> and predict the action -> return the action
Comments.UserDisplayName: Sanyou
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-08T01:30:07.090Z
Comments.Text: Yes, I think actor’s state_dict should be passed, I’ve just tried to pass the agent and I got the same error. Let me give a try. Thanks.
Comments.UserDisplayName: Mika
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-08T01:40:12.997Z
Comments.Text: if you're on linux try other start method (spawn, fork, forkserver) and see the effect on the running time, or even start all processes first and send the state_dict and state to Queue to prevent instantiating processes on every step.
Comments.UserDisplayName: Sanyou
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-03T13:15:37.593Z
Comments.Text: @Mika were you able to solve this problem? Any help would be much appreciated
Comments.UserDisplayName: Gopal Chitalia
Tags: <reinforcement-learning><python><deep-rl><ddpg><multi-agent-systems>
Title: How to parallelize multi-agent DDPG (MADDPG)
UserDisplayName: Mika
ThreadText: # How to parallelize multi-agent DDPG (MADDPG)
  
  (Asked by: username_0 on 2021-09-05)
  
  I am experimenting with MADDPG algorithm implemented in this repo. Since there were only a few agents (2-3) in the implementation (also in the original paper) steps like parameter updates, action prediction, etc. are done in a for loop. I want to increase the number of agents, say 10 or 30, and perform parallelization of the above-mentioned steps for all agents, i.e. I want to avoid for loops like this
  for agent_idx in range(n_agents):
      ...
      ...
  
  I tried Python Multiprocessing module with pool.map method but I am getting 'AttributeError: Can't pickle local object ...". Below is code I am running to get a joint action prediction but it results in the error above.
  def get_ind_action(i, obs_i):
      return actor_critic[i].act(obs_i) # returns an individual action for a given observation for ith agent
  
  def get_joint_action(obs):
      pool = Pool()
      args_list = [[i, obs[i]] for i in range(n_agents)]
      joint_action = pool.map(get_ind_action, args_list)
      return joint_action
      
  
  Here actor_critic is a list of neural networks of all agents, obs is the joint state observed by the centralized critic but each actor only sees its own state. The algorithm has the following architecture.
  
  **Tags:** reinforcement-learning, python, deep-rl, ddpg, multi-agent-systems
  > **Comments:**
  > **username_1**: Hey, maybe to help someone unfamiliar with MADDPG (like me) to quickly grasp the difficulty of parallelizing it can you clarify that (a) are the agents gonna share info about their parameters at some point? (b) and if any other information is going to be shared among process in the learning phase? (c) are you perhaps planning to parallelize on single node or multi nodes? (d) to answer the error perhaps share short reproducible parts of the code that you add to the code you linked  
  > **username_0**: (a) they have identical architecture, i.e. the same number of layers and units in each layer but share no parameters. (b) no information is shared among agents' actor networks. (c) single node. (d) added above  
  > **username_1**: You have to pass the agent as the argument to the function, because the subprocess do not have the agent in the memory. You might want to pass the actor's state_dict instead of the agent and reinstantiate the agent in the subprocess->load the state_dict -> and predict the action -> return the action  
  > **username_0**: Yes, I think actor’s state_dict should be passed, I’ve just tried to pass the agent and I got the same error. Let me give a try. Thanks.  
  > **username_1**: if you're on linux try other start method (spawn, fork, forkserver) and see the effect on the running time, or even start all processes first and send the state_dict and state to Queue to prevent instantiating processes on every step.  
  > **username_2**: @Mika were you able to solve this problem? Any help would be much appreciated  
  
  ---
  
  ## 0 answers
  
  