Answers.Body: <p>As long as your policy (propensity) is differentiable, everything's is good. Discrete, continuous, other, doesn't matter! :)</p>
  <p>A common example for continuous spaces is the <a href="https://spinningup.readthedocs.io/zh_CN/latest/algorithms/sac.html" rel="nofollow noreferrer">reparameterization trick</a>, where your policy outputs <span class="math-container">$\mu, \sigma = \pi(s)$</span> and the action is <span class="math-container">$a \sim \mathcal{N}(\mu, \sigma)$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-12T14:03:55.783Z
Answers.Text: Thanks for your answer! So, as I see it, action values are indeed sampled from some distribution in both the continuous and discrete case. In the continuous case, always(?) a Gaussian distribution is used of which the parameters are predicted. But how does the distribution look like for the discrete case? Is it indeed that in the discrete case, probabilities for sampling actions are only given by (possibly non-Gaussian-like distributed) probability vectors? And is there any nice paper that officially introduced these sampling schemes?
Answers.UserDisplayName: Daniel B.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-12T21:19:06.147Z
Answers.Text: In the discrete case, you can do epsilon greedy, softmax, or anything really. As long as the density of your policy is differentiable, you can run PPO
Answers.UserDisplayName: kaiwenw
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-13T02:04:50.683Z
Answers.Text: So, after all there's no such thing like a fixed output layer architecture that a deep RL model would have to use in order to qualify as a PPO variant? It's really up to the researcher to decide how he/she want's a PPO variant's policy network's output to look like? Just double-checking one more time. :)
Answers.UserDisplayName: Daniel B.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-13T05:59:33.277Z
Answers.Text: @DanielB. exactly! :) the essence of REINFORCE, PPO, TRPO, Q-learning are the *way* the actors are updated, rather than a specific deep network architecture. For example, PPO/TRPO tries to stay in a "Trust Region", regardless of what policy architecture you choose.
Answers.UserDisplayName: kaiwenw
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-13T10:56:34.540Z
Answers.Text: Ok, thank you very much! I think I am just subconsciously a bit biased towards believing that one always has to have a certain kind of output layer architecture, since I previously worked with Q-learning where it is of course pretty much dictated that, however the output layer looks like, it must be able to predict arbitrary Q-values (which of course makes the use certain activation functions etc impractical) and where the meaning of the output is clearly predetermined (to be Q-values), whereas the interpretation of outputs in PPO, TRPO... seems to be much more flexible.
Answers.UserDisplayName: Daniel B.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-14T12:06:12.340Z
Answers.Text: In the continuous case, how would the probability $\pi_\theta(a_t|s_t)$ be computed since we don't predict probability vectors any longer, but unconstrained real numbers instead? Just asking because we still need this to be able to compute the probability ratio $r(\theta)$.
Answers.UserDisplayName: Daniel B.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-04T11:11:03.747Z
Answers.Text: @DanielB.  It seems that in the continuous case one has to use the re-parameterization trick or other methods to deal with probability distributions.  I'm not sure if the re-parameterization trick allows multi-modal probability distributions to be represented.  This seems to be a complicated topic.
Answers.UserDisplayName: Yan King Yin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-12T06:22:09.017Z
Answers.LastActivityDate: 2020-12-12T12:05:16.137Z
Answers.LastEditDate: 2020-12-12T12:05:16.137Z
Answers.UserDisplayName: kaiwenw
Body: <p>I am trying to understand and reproduce the <strong><a href="https://arxiv.org/pdf/1707.06347.pdf" rel="noreferrer">Proximal Policy Optimization (PPO)</a></strong> algorithm in detail.  One thing that I find missing in the <a href="https://arxiv.org/pdf/1707.06347.pdf" rel="noreferrer">paper</a> introducing the algorithm is how exactly actions <span class="math-container">$a_t$</span> are generated given the policy network <span class="math-container">$\pi_\theta(a_t|s_t)$</span>.</p>
  <p>From the <a href="https://github.com/openai/baselines/blob/master/baselines/ppo1/pposgd_simple.py#L32" rel="noreferrer">source code</a>, I saw that <em>discrete</em> actions get sampled from some probability distribution (which I assume to be discrete in this case) parameterized by the output probabilities generated by <span class="math-container">$\pi_\theta$</span> given the state <span class="math-container">$s_t$</span>.</p>
  <p>However, what I don't understand is how <em>continuous</em> actions are sampled/generated from the policy network. Are they also sampled from a (probably continuous) distribution? In that case, which type of distribution is used and which parameters are predicted by the policy network to parameterize said distribution?</p>
  <p>Also, is there any official literature that I could cite which introduces the method by which PPO generates its action outputs?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-16T19:35:42.820Z
Comments.Text: If you have a new question, you should ask it in a different post, even though it's related to the current question (I actually don't know), because that may invalidate the existing answers.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-16T20:44:22.880Z
Comments.Text: I think these answers I am searching for belong fundamentally together since only knowing what to predict does not make sense in absence of the knowledge about how to get to that prediction eventually. And for getting there, the surrogate loss needs to be considered as well since otherwise you don't have any way to properly train the model (in spite of knowing what it shall predict). And just for the context: $r_t(\theta)$ is an important part of the surrogate loss. But anyway. I sort of see your point, so let's make it a separate question then.
Comments.UserDisplayName: Daniel B.
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-16T20:54:20.490Z
Comments.Text: Never mind. I think this [question](https://ai.stackexchange.com/q/23276/37982) contains actually the answer to my edited question. I only hope that it is correct because this suspected answer is phrased as part of a question. But it sounds like a reasonable approach. So sorry for the confusion & I will 'revert' the edit.
Comments.UserDisplayName: Daniel B.
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-16T21:16:01.597Z
Comments.Text: If you're not satisfied with that answer, eventually, you could ask a similar question but make sure to provide the context and say why you're not satisfied with that answer.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><implementation><proximal-policy-optimization><continuous-action-spaces>
Title: How are continuous actions sampled (or generated) from the policy network in PPO?
UserDisplayName: Daniel B.
ThreadText: # How are continuous actions sampled (or generated) from the policy network in PPO?
  
  (Asked by: username_0 on 2020-12-12)
  
  I am trying to understand and reproduce the **[Proximal Policy Optimization (PPO)](https://arxiv.org/pdf/1707.06347.pdf)** algorithm in detail. One thing that I find missing in the [paper](https://arxiv.org/pdf/1707.06347.pdf) introducing the algorithm is how exactly actions $a\_t$ are generated given the policy network $\pi\_\theta(a\_t|s\_t)$.
  
  From the [source code](https://github.com/openai/baselines/blob/master/baselines/ppo1/pposgd_simple.py#L32), I saw that *discrete* actions get sampled from some probability distribution (which I assume to be discrete in this case) parameterized by the output probabilities generated by $\pi\_\theta$ given the state $s\_t$.
  
  However, what I don't understand is how *continuous* actions are sampled/generated from the policy network. Are they also sampled from a (probably continuous) distribution? In that case, which type of distribution is used and which parameters are predicted by the policy network to parameterize said distribution?
  
  Also, is there any official literature that I could cite which introduces the method by which PPO generates its action outputs?
  
  **Tags:** reinforcement-learning, implementation, proximal-policy-optimization, continuous-action-spaces
  
  > **Comments:**
  > **username_1**: If you have a new question, you should ask it in a different post, even though it's related to the current question (I actually don't know), because that may invalidate the existing answers.  
  > **username_0**: I think these answers I am searching for belong fundamentally together since only knowing what to predict does not make sense in absence of the knowledge about how to get to that prediction eventually. And for getting there, the surrogate loss needs to be considered as well since otherwise you don't have any way to properly train the model (in spite of knowing what it shall predict). And just for the context: $r_t(\theta)$ is an important part of the surrogate loss. But anyway. I sort of see your point, so let's make it a separate question then.  
  > **username_0**: Never mind. I think this [question](https://ai.stackexchange.com/q/23276/37982) contains actually the answer to my edited question. I only hope that it is correct because this suspected answer is phrased as part of a question. But it sounds like a reasonable approach. So sorry for the confusion & I will 'revert' the edit.  
  > **username_1**: If you're not satisfied with that answer, eventually, you could ask a similar question but make sure to provide the context and say why you're not satisfied with that answer.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-12-12. Score: 3 | Accepted answer
  
  As long as your policy (propensity) is differentiable, everything's is good. Discrete, continuous, other, doesn't matter! :)
  
  A common example for continuous spaces is the [reparameterization trick](https://spinningup.readthedocs.io/zh_CN/latest/algorithms/sac.html), where your policy outputs $\mu, \sigma = \pi(s)$ and the action is $a \sim \mathcal{N}(\mu, \sigma)$.
  
  > **Comments:**
  > **username_0**: Thanks for your answer! So, as I see it, action values are indeed sampled from some distribution in both the continuous and discrete case. In the continuous case, always(?) a Gaussian distribution is used of which the parameters are predicted. But how does the distribution look like for the discrete case? Is it indeed that in the discrete case, probabilities for sampling actions are only given by (possibly non-Gaussian-like distributed) probability vectors? And is there any nice paper that officially introduced these sampling schemes?  
  > **username_2**: In the discrete case, you can do epsilon greedy, softmax, or anything really. As long as the density of your policy is differentiable, you can run PPO  
  > **username_0**: So, after all there's no such thing like a fixed output layer architecture that a deep RL model would have to use in order to qualify as a PPO variant? It's really up to the researcher to decide how he/she want's a PPO variant's policy network's output to look like? Just double-checking one more time. :)  
  > **username_2**: @DanielB. exactly! :) the essence of REINFORCE, PPO, TRPO, Q-learning are the *way* the actors are updated, rather than a specific deep network architecture. For example, PPO/TRPO tries to stay in a "Trust Region", regardless of what policy architecture you choose.  
  > **username_0**: Ok, thank you very much! I think I am just subconsciously a bit biased towards believing that one always has to have a certain kind of output layer architecture, since I previously worked with Q-learning where it is of course pretty much dictated that, however the output layer looks like, it must be able to predict arbitrary Q-values (which of course makes the use certain activation functions etc impractical) and where the meaning of the output is clearly predetermined (to be Q-values), whereas the interpretation of outputs in PPO, TRPO... seems to be much more flexible.  
  > **username_0**: In the continuous case, how would the probability $\pi_\theta(a_t|s_t)$ be computed since we don't predict probability vectors any longer, but unconstrained real numbers instead? Just asking because we still need this to be able to compute the probability ratio $r(\theta)$.  
  > **username_3**: @DanielB.  It seems that in the continuous case one has to use the re-parameterization trick or other methods to deal with probability distributions.  I'm not sure if the re-parameterization trick allows multi-modal probability distributions to be represented.  This seems to be a complicated topic.  
  
  ---
  
  