Answers.Body: <p>A natural policy to act in an environment with discrete action space would be a softmax.</p>
  <p>This <a href="https://arxiv.org/abs/1706.02275" rel="nofollow noreferrer">paper</a> describes a method that uses the idea of centralized training, and I believe could be used in your implementation.</p>
  <p>With regard to your last question, I don't know if i understood, but if you have a system that must perform 3 actions, you could assign each action to a specific agent (assuming we have three different action spaces). Then you would have a cooperation game with 3 agents, where all of them have a common reward function. In theory, this 3 agents represents an individual agent that interacts with the environment.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-18T17:04:52.330Z
Answers.LastActivityDate: 2021-05-18T17:04:52.330Z
Answers.UserDisplayName: Felipe Costa
Body: <p>Is there a multi-agent deep reinforcement learning algorithm which is for environments with only discrete action spaces (Not hybrid) and have centralized training?</p>
  <p>I have been looking for algorithms, (A2C, MADDPG etc.) but still havent find any algorithm that provides all of properties i mentioned (Multi agent + discrete action space + deep learning + centralized training).</p>
  <p>I am wondering if we use an actor network that gets state as input and concatenated discrete actions of agents as output (For example if agent has 3 actions and we have 4 agents output can be [0,0,1, 0,1,0, 0,0,1, 1,0,0]) is that would be bad idea ?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-17T13:01:54.627Z
Comments.Text: I havn't done RL in a while, but isn't the point of multi agent the fact that they act on their own and thus are disctinct networks ? Why would you use multi agent if you merge them into one network agent ? I remember [Unity ML-Agents](https://github.com/Unity-Technologies/ml-agents/blob/main/docs/ML-Agents-Overview.md) plugin providing this kind of multi agent learning, maybe you could have a look at how they do it.
Comments.UserDisplayName: Ubikuity
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-18T16:45:57.837Z
Comments.Text: @Ubikuity if the agents are homogeneous, then you can have a common NN policy that is executed decentralized but have centralized training. Also, the last time I worked with Unity ML-Agents they didn't have a direct implementation of MARL algorithms. As I remember they have the usual RL PPO and SAC algorithms, although they can work in simple MARL implementations.
Comments.UserDisplayName: Felipe Costa
Tags: <reinforcement-learning><deep-rl><discrete-action-spaces><multi-agent-rl>
Title: Is there a multi-agent deep reinforcement learning algorithm which is for environments with only discrete action spaces (Not hybrid)?
UserDisplayName: Uur Kn
ThreadText: # Is there a multi-agent deep reinforcement learning algorithm which is for environments with only discrete action spaces (Not hybrid)?
  
  (Asked by: username_0 on 2021-05-17)
  
  Is there a multi-agent deep reinforcement learning algorithm which is for environments with only discrete action spaces (Not hybrid) and have centralized training?
  
  I have been looking for algorithms, (A2C, MADDPG etc.) but still havent find any algorithm that provides all of properties i mentioned (Multi agent + discrete action space + deep learning + centralized training).
  
  I am wondering if we use an actor network that gets state as input and concatenated discrete actions of agents as output (For example if agent has 3 actions and we have 4 agents output can be [0,0,1, 0,1,0, 0,0,1, 1,0,0]) is that would be bad idea ?
  
  **Tags:** reinforcement-learning, deep-rl, discrete-action-spaces, multi-agent-rl
  
  > **Comments:**
  > **username_1**: I havn't done RL in a while, but isn't the point of multi agent the fact that they act on their own and thus are disctinct networks ? Why would you use multi agent if you merge them into one network agent ? I remember [Unity ML-Agents](https://github.com/Unity-Technologies/ml-agents/blob/main/docs/ML-Agents-Overview.md) plugin providing this kind of multi agent learning, maybe you could have a look at how they do it.  
  > **username_2**: @username_1 if the agents are homogeneous, then you can have a common NN policy that is executed decentralized but have centralized training. Also, the last time I worked with Unity ML-Agents they didn't have a direct implementation of MARL algorithms. As I remember they have the usual RL PPO and SAC algorithms, although they can work in simple MARL implementations.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-05-18. Score: 1
  
  A natural policy to act in an environment with discrete action space would be a softmax.
  
  This [paper](https://arxiv.org/abs/1706.02275) describes a method that uses the idea of centralized training, and I believe could be used in your implementation.
  
  With regard to your last question, I don't know if i understood, but if you have a system that must perform 3 actions, you could assign each action to a specific agent (assuming we have three different action spaces). Then you would have a cooperation game with 3 agents, where all of them have a common reward function. In theory, this 3 agents represents an individual agent that interacts with the environment.
  
  
  ---
  
  