Answers.Body: <p>Seems like what you are looking for is Parametrized RL to train an agent for a Parametrized Markov Decision Process. You can look up both terms to find courses/readings about it.</p>
  <p>Anyway, one existing RL framework for it is the <a href="https://github.com/cycraig/MP-DQN" rel="nofollow noreferrer">Multi-pass Parametrized DQN (MPDQN)</a> which is proposed in <a href="https://arxiv.org/abs/1905.04388" rel="nofollow noreferrer">this paper</a>, and if google enoug you can even find the author's dissertation on MP-DQN which preliminaries section might help you a lot. In short, the Agent uses Actor-Critic arhitecture in which the Actor will predict the continuous parameters of all actions given the current state, and the Critic will predict the action-value of each action given the states concatenated by the predicted values of the continuous parameters. The final discrete action is chosen by sampling the action-value or using argmax.</p>
  <p>In <a href="https://arxiv.org/abs/1810.06394" rel="nofollow noreferrer">P-DQN</a>, the state concatenated with all the continuous parameters' values are passed to the Critic. However, it is shown that by passing all continuous parameters at once to the Critic, the unused action's parameters will affect the gradient thus affecting the Actor's parameters, which is not expected. Therefore, the parameters of each action are passed separately into the Critic in MP-DQN (unrelated action parameters' are zeroed). With good design, you can implement it so you can pass all the parameters and state in a single pass (multi-pass) as shown in MP-DQN. Other alternative for Parametrized RL is the <a href="https://www.ijcai.org/proceedings/2019/0316.pdf" rel="nofollow noreferrer">hybrid-PPO</a> and <a href="https://arxiv.org/abs/1511.04143" rel="nofollow noreferrer">parametrized DDPG</a>.</p>
  <p>Finally, based on my personal experiments, I still have difficulty on controlling the scale of the continuous parameters values prediction so that it stays in a certain predefined range. There are 2 ways to do it, first is using softmax or using gradient inverting (changing the gradients' sign depending on the current predicted parameter values). However, I still eventually produced Actor that output only extreme values (either upper or lower bound) of the continuous parameters.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-17T21:01:18.570Z
Answers.Text: Hi sanyou I have been working my way through the implementation of MPDQN in the github and it seems like they only handle the case where there is one discrete action possible and then many continuous actions.  I am in the case where I have a multi-discrete space and continuous actions.  Do you think this method would be able to handle this?
Answers.UserDisplayName: pd109
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-18T05:34:54.493Z
Answers.Text: yes it can, if you look closely at the agent class definition, there's an intricate masking operation to differentiate which continuous parameters are related to which discrete action.
Answers.UserDisplayName: Sanyou
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-20T22:22:18.133Z
Answers.Text: I can see how they are handling the parameterized action spaces in `env.action_space ` where the first space is a Discrete that describes the total number of actions and then the rest are box spaces that hold the high and low value information.  However I am not certain how to define my action space in these terms since I have some actions that are just completely discrete.  For example I have one action that is just a discrete action between 5 choices and then also 2 other actions that are continuous.  All of these actions can be used at a step together.
Answers.UserDisplayName: pd109
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-21T06:16:00.663Z
Answers.Text: for the discrete action, just don't use any of the continuous parameter, so that no grad will be backpropagated. While for the continuous actions, that's what the discrete action/label is for, to point what action you choose, and you just so happen to choose a continuous action. Note that, while the continuous actor give you real values, the discrete critic just give you which action to do, right? so no problem with the continuous actions.
Answers.UserDisplayName: Sanyou
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-11T02:47:02.070Z
Answers.LastActivityDate: 2021-09-11T02:47:02.070Z
Answers.UserDisplayName: Sanyou
Body: <p>I am looking to have a cooperative multi agent reinforcement learning framework where one agent has a discrete action space and another agent has a continuous action space. Is there a way to do this as most papers I have seen will only handle one or the other.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-10T03:24:00.700Z
Comments.Text: you might want to google "parametrized action reinforcement learning" and "parametrized Markov decision process"
Comments.UserDisplayName: Sanyou
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-10T17:49:42.160Z
Comments.Text: Thanks Sanyou seems like what I have been looking for.  I just didn't really know the correct terms to google were
Comments.UserDisplayName: pd109
Tags: <deep-learning><deep-rl>
Title: Multi Agent Deep Reinforcement Learning for continuous and discrete action
UserDisplayName: pd109
ThreadText: # Multi Agent Deep Reinforcement Learning for continuous and discrete action
  
  (Asked by: username_0 on 2021-09-09)
  
  I am looking to have a cooperative multi agent reinforcement learning framework where one agent has a discrete action space and another agent has a continuous action space. Is there a way to do this as most papers I have seen will only handle one or the other.
  
  **Tags:** deep-learning, deep-rl
  > **Comments:**
  > **username_1**: you might want to google "parametrized action reinforcement learning" and "parametrized Markov decision process"  
  > **username_0**: Thanks Sanyou seems like what I have been looking for.  I just didn't really know the correct terms to google were  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-09-11. Score: 1 | Accepted answer
  
  Seems like what you are looking for is Parametrized RL to train an agent for a Parametrized Markov Decision Process. You can look up both terms to find courses/readings about it.
  
  Anyway, one existing RL framework for it is the [Multi-pass Parametrized DQN (MPDQN)](https://github.com/cycraig/MP-DQN) which is proposed in [this paper](https://arxiv.org/abs/1905.04388), and if google enoug you can even find the author's dissertation on MP-DQN which preliminaries section might help you a lot. In short, the Agent uses Actor-Critic arhitecture in which the Actor will predict the continuous parameters of all actions given the current state, and the Critic will predict the action-value of each action given the states concatenated by the predicted values of the continuous parameters. The final discrete action is chosen by sampling the action-value or using argmax.
  
  In [P-DQN](https://arxiv.org/abs/1810.06394), the state concatenated with all the continuous parameters' values are passed to the Critic. However, it is shown that by passing all continuous parameters at once to the Critic, the unused action's parameters will affect the gradient thus affecting the Actor's parameters, which is not expected. Therefore, the parameters of each action are passed separately into the Critic in MP-DQN (unrelated action parameters' are zeroed). With good design, you can implement it so you can pass all the parameters and state in a single pass (multi-pass) as shown in MP-DQN. Other alternative for Parametrized RL is the [hybrid-PPO](https://www.ijcai.org/proceedings/2019/0316.pdf) and [parametrized DDPG](https://arxiv.org/abs/1511.04143).
  
  Finally, based on my personal experiments, I still have difficulty on controlling the scale of the continuous parameters values prediction so that it stays in a certain predefined range. There are 2 ways to do it, first is using softmax or using gradient inverting (changing the gradients' sign depending on the current predicted parameter values). However, I still eventually produced Actor that output only extreme values (either upper or lower bound) of the continuous parameters.
  
  > **Comments:**
  > **username_0**: Hi sanyou I have been working my way through the implementation of MPDQN in the github and it seems like they only handle the case where there is one discrete action possible and then many continuous actions.  I am in the case where I have a multi-discrete space and continuous actions.  Do you think this method would be able to handle this?  
  > **username_1**: yes it can, if you look closely at the agent class definition, there's an intricate masking operation to differentiate which continuous parameters are related to which discrete action.  
  > **username_0**: I can see how they are handling the parameterized action spaces in `env.action_space ` where the first space is a Discrete that describes the total number of actions and then the rest are box spaces that hold the high and low value information.  However I am not certain how to define my action space in these terms since I have some actions that are just completely discrete.  For example I have one action that is just a discrete action between 5 choices and then also 2 other actions that are continuous.  All of these actions can be used at a step together.  
  > **username_1**: for the discrete action, just don't use any of the continuous parameter, so that no grad will be backpropagated. While for the continuous actions, that's what the discrete action/label is for, to point what action you choose, and you just so happen to choose a continuous action. Note that, while the continuous actor give you real values, the discrete critic just give you which action to do, right? so no problem with the continuous actions.  
  
  ---
  
  