Answers.Body: <p>As you has said, actions chosen by Actor-Critic typically come from a normal distribution and it is the agent's job to find the appropriate mean and standard deviation based on the the current state. In many cases this one distribution is enough because only 1 continuous action is required. However, as domains such as robotics become more integrated with AI, situations where 2 or more continuous actions are required are a growing problem.</p>
  
  <p>There are 2 solutions to this problem:
  The first and most common is that for every continuous action, there is a separate agent learning its own 1-dimensional mean and standard deviation. Part of its state includes the actions of the other agents as well to give context of what the entire system is doing. We commonly do this in my lab and <a href="https://sites.ualberta.ca/~pilarski/docs/papers/Pilarski_2013_ICORR_Postprint.pdf" rel="nofollow noreferrer">here is a paper which describes this approach</a> with 3 actor-critic agents working together to move a robotic arm. </p>
  
  <p>The second approach is to have one agent find a <a href="https://en.wikipedia.org/wiki/Multivariate_normal_distribution" rel="nofollow noreferrer">multivariate (usually normal) distribution</a> of a policy. Although in theory, this approach could have a more concise policy distribution by "rotating" the distribution based on the co-variance matrix, it means that all of the values of the co-variance matrix must be learned as well. This increases the number of values that must be learned to have $n$ continuous outputs from $2n$ (mean and stddev), to $n+n^2$ ($n$ means and an $n \times n$ co-variance matrix). This drawback has made this approach not as popular in the literature.</p>
  
  <p>This is a more general answer but should help you and others on their related problems.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-09-22T08:48:08.820Z
Answers.Text: Jaden thanks for great answer. 
  1. I tried multi-agent architecture, but it is not very efficient. Takes much longer to converge.
  2. Now multivariate distribution seems obvious to me too, thank you.
Answers.UserDisplayName: Evalds Urtans
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-09-22T13:14:58.357Z
Answers.Text: Depending on the application and architecture (if it is a deep net), you can have the agents share low level features and then have them branch off into their own value functions. Additionally, having 1 critic and multiple actors is also a way to increase the architecture.
Answers.UserDisplayName: Jaden Travnik
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-09-25T11:55:27.430Z
Answers.Text: At the moment I would like to apply your suggestions to TRPO (just policy gradient methods), not actor-critic. I am not very confident in gradient transfer from critic to actor - in many implementations I have seen it looks like that it should not work even though it does converge.
Answers.UserDisplayName: Evalds Urtans
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-02-12T07:13:20.657Z
Answers.Text: Sorry for this noob question: How is this applied in actor-critic methods (where the actor can perform multiple simultaneous continuous actions), where the actor has the policy function and gets trained by policy gradient method? @JadenTravnik Can you please explain that in the answer under a new heading?
Answers.UserDisplayName: Gokul NC
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-09-22T00:53:52.963Z
Answers.LastActivityDate: 2018-08-19T12:10:13.233Z
Answers.LastEditDate: 2018-08-19T12:10:13.233Z
Answers.UserDisplayName: Jaden Travnik
Body: <p>Trusted Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) are two cutting edge policy gradients algorithms.</p>
  <p>When using a single continuous action, normally, you would use some probability distribution (for example, Gaussian) for the loss function. The rough version is:</p>
  <p><span class="math-container">$$L(\theta) = \log(P(a_1)) A,$$</span></p>
  <p>where <span class="math-container">$A$</span> is the advantage of rewards, <span class="math-container">$P(a_1)$</span> is characterized by <span class="math-container">$\mu$</span> and <span class="math-container">$\sigma^2$</span> that comes out of neural network like in the Pendulum environment here: <a href="https://github.com/leomzhong/DeepReinforcementLearningCourse/blob/69e573cd88faec7e9cf900da8eeef08c57dec0f0/hw4/main.py" rel="nofollow noreferrer">https://github.com/leomzhong/DeepReinforcementLearningCourse/blob/69e573cd88faec7e9cf900da8eeef08c57dec0f0/hw4/main.py</a>.</p>
  <p>The problem is that I cannot find any paper on 2+ continuous actions using policy gradients (not actor-critic methods that use a different approach by transferring gradient from Q-function).</p>
  <p>Do you know how to do this using TRPO for 2 continuous actions in <a href="https://gym.openai.com/envs/LunarLanderContinuous-v2/" rel="nofollow noreferrer">LunarLander environment</a>?</p>
  <p>Is following approach correct for policy gradient loss function?</p>
  <p><span class="math-container">$$L(\theta) = (\log P(a_1) + \log P(a_2) )*A$$</span></p>
  
Tags: <deep-learning><reinforcement-learning><policy-gradients><proximal-policy-optimization><trust-region-policy-optimization>
Title: How can policy gradients be applied in the case of multiple continuous actions?
UserDisplayName: Evalds Urtans
ThreadText: # How can policy gradients be applied in the case of multiple continuous actions?
  
  (Asked by: username_0 on 2017-09-21)
  
  Trusted Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) are two cutting edge policy gradients algorithms.
  
  When using a single continuous action, normally, you would use some probability distribution (for example, Gaussian) for the loss function. The rough version is:
  
  $$L(\theta) = \log(P(a\_1)) A,$$
  
  where $A$ is the advantage of rewards, $P(a\_1)$ is characterized by $\mu$ and $\sigma^2$ that comes out of neural network like in the Pendulum environment here: <https://github.com/leomzhong/DeepReinforcementLearningCourse/blob/69e573cd88faec7e9cf900da8eeef08c57dec0f0/hw4/main.py>.
  
  The problem is that I cannot find any paper on 2+ continuous actions using policy gradients (not actor-critic methods that use a different approach by transferring gradient from Q-function).
  
  Do you know how to do this using TRPO for 2 continuous actions in [LunarLander environment](https://gym.openai.com/envs/LunarLanderContinuous-v2/)?
  
  Is following approach correct for policy gradient loss function?
  
  $$L(\theta) = (\log P(a\_1) + \log P(a\_2) )\*A$$
  
  **Tags:** deep-learning, reinforcement-learning, policy-gradients, proximal-policy-optimization, trust-region-policy-optimization
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2017-09-22. Score: 8 | Accepted answer
  
  As you has said, actions chosen by Actor-Critic typically come from a normal distribution and it is the agent's job to find the appropriate mean and standard deviation based on the the current state. In many cases this one distribution is enough because only 1 continuous action is required. However, as domains such as robotics become more integrated with AI, situations where 2 or more continuous actions are required are a growing problem.
  
  There are 2 solutions to this problem:
  The first and most common is that for every continuous action, there is a separate agent learning its own 1-dimensional mean and standard deviation. Part of its state includes the actions of the other agents as well to give context of what the entire system is doing. We commonly do this in my lab and [here is a paper which describes this approach](https://sites.ualberta.ca/~pilarski/docs/papers/Pilarski_2013_ICORR_Postprint.pdf) with 3 actor-critic agents working together to move a robotic arm.
  
  The second approach is to have one agent find a [multivariate (usually normal) distribution](https://en.wikipedia.org/wiki/Multivariate_normal_distribution) of a policy. Although in theory, this approach could have a more concise policy distribution by "rotating" the distribution based on the co-variance matrix, it means that all of the values of the co-variance matrix must be learned as well. This increases the number of values that must be learned to have $n$ continuous outputs from $2n$ (mean and stddev), to $n+n^2$ ($n$ means and an $n \times n$ co-variance matrix). This drawback has made this approach not as popular in the literature.
  
  This is a more general answer but should help you and others on their related problems.
  
  > **Comments:**
  > **username_0**: Jaden thanks for great answer.  1. I tried multi-agent architecture, but it is not very efficient. Takes much longer to converge. 2. Now multivariate distribution seems obvious to me too, thank you.  
  > **username_1**: Depending on the application and architecture (if it is a deep net), you can have the agents share low level features and then have them branch off into their own value functions. Additionally, having 1 critic and multiple actors is also a way to increase the architecture.  
  > **username_0**: At the moment I would like to apply your suggestions to TRPO (just policy gradient methods), not actor-critic. I am not very confident in gradient transfer from critic to actor - in many implementations I have seen it looks like that it should not work even though it does converge.  
  > **username_2**: Sorry for this noob question: How is this applied in actor-critic methods (where the actor can perform multiple simultaneous continuous actions), where the actor has the policy function and gets trained by policy gradient method? @JadenTravnik Can you please explain that in the answer under a new heading?  
  
  ---
  
  