Answers.Body: <p>First of all, the support of a normal distribution is the entire real line (or, in general, <span class="math-container">$\mathbb{R}^n$</span> for an <span class="math-container">$n$</span>-dimensional multivariate normal distribution) so your action can be any number in <span class="math-container">$\mathbb{R}$</span>. What you may be getting confused with is that with probability 0.68 you will obtain an action that is within +/- 1 standard deviation from the mean.</p>
  <p>Now, to answer the question of how you can do this using RL:</p>
  <p>To use the Normal distribution in this setting I would simply clip my actions in the environment. If, for example, the actor gives an action below your minimum value <span class="math-container">$X$</span>, lets say it gives <span class="math-container">$X-0.5$</span>, then I would simply clip the action to be <span class="math-container">$X$</span> when executed in the environment. This way your actor can still sample from a normal distribution which could give answers below <span class="math-container">$X$</span> (or above <span class="math-container">$Y$</span>) and be used with your environment.</p>
  <p>If, for instance, your desired range was <span class="math-container">$(-1, 1)$</span> then another option would be to define your distribution to be <span class="math-container">$Y = \mbox{tanh}(X)$</span>, where <span class="math-container">$X \sim N(\mu, \sigma)$</span>. You can then find the density function of <span class="math-container">$Y$</span> using e.g. the <a href="https://en.wikipedia.org/wiki/Probability_density_function#Function_of_random_variables_and_change_of_variables_in_the_probability_density_function" rel="nofollow noreferrer">density transformation method</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-09T09:08:33.927Z
Answers.Text: I see, thank you so much for clearing that about the normal distribution. From what I understand, the normal distribution is just a special case of the Gaussian distribution? Is that correct?
  
  What if, for example, the output of the actor leads to an action range that is much smaller than the actual range of actions. That is, what if the range of output is [-2, 2] but the actual action range is [-5, 5]. Is expanding the output (unsquashing it) into the original action range of [-5,5] a valid thing to do?
Answers.UserDisplayName: hridayns
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-09T10:16:33.513Z
Answers.Text: the normal distribution _is_ the Gaussian distribution, it is just a different name that people use :) in that instance, I would recommend to do some kind of normalisation so that the actors range equals the actual environments range. If the actual environment range is symmetric, i.e. $[-x, x]$ then you could use a `tanh` activation function for the final layer of the network and multiply the output by $x$.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-09T10:28:32.163Z
Answers.Text: Yeah, sadly my action range isn't symmetric! But thanks for the help!
Answers.UserDisplayName: hridayns
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-09T10:48:36.340Z
Answers.Text: you could try shifting you action range to start from 0, i.e. if the range is $[x, y]$ then make the range $[0, y-x]$ and then you just need to use a `relu` activation and clip at $y-x$, or use a `sigmoid` and multiply by $y-x$.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-08T10:08:19.807Z
Answers.LastActivityDate: 2021-10-19T12:35:49.787Z
Answers.LastEditDate: 2021-10-19T12:35:49.787Z
Answers.UserDisplayName: David
Body: <p>Specifically for <strong>continuous control PPO</strong>, let's say my action space range is between  <strong><span class="math-container">$X$</span> (low) and <span class="math-container">$Y$</span> (high)</strong> and they are all sampled from a <strong>Gaussian Action Distribution</strong> with mean <strong><span class="math-container">$\mu$</span></strong> and standard deviation <strong><span class="math-container">$\rho$</span></strong>.</p>
  <p>From what I understood, the actions sampled should fall between <strong><span class="math-container">$\mu - \rho$</span></strong> and <strong><span class="math-container">$\mu + \rho$</span></strong>, but that's not what happens in practice? What am I misunderstanding here? How do I ensure this range constraint from a custom action distribution with a given mean and standard deviation?</p>
  <p>Any advice or tips for me? I would really appreciate any insights!</p>
  
Tags: <reinforcement-learning><probability-distribution><proximal-policy-optimization><continuous-action-spaces><normal-distribution>
Title: How to define a continuous action distribution with a specific range for Reinforcement Learning?
UserDisplayName: hridayns
ThreadText: # How to define a continuous action distribution with a specific range for Reinforcement Learning?
  
  (Asked by: username_0 on 2021-07-07)
  
  Specifically for **continuous control PPO**, let's say my action space range is between **$X$ (low) and $Y$ (high)** and they are all sampled from a **Gaussian Action Distribution** with mean **$\mu$** and standard deviation **$\rho$**.
  
  From what I understood, the actions sampled should fall between **$\mu - \rho$** and **$\mu + \rho$**, but that's not what happens in practice? What am I misunderstanding here? How do I ensure this range constraint from a custom action distribution with a given mean and standard deviation?
  
  Any advice or tips for me? I would really appreciate any insights!
  
  **Tags:** reinforcement-learning, probability-distribution, proximal-policy-optimization, continuous-action-spaces, normal-distribution
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-07-08. Score: 1 | Accepted answer
  
  First of all, the support of a normal distribution is the entire real line (or, in general, $\mathbb{R}^n$ for an $n$-dimensional multivariate normal distribution) so your action can be any number in $\mathbb{R}$. What you may be getting confused with is that with probability 0.68 you will obtain an action that is within +/- 1 standard deviation from the mean.
  
  Now, to answer the question of how you can do this using RL:
  
  To use the Normal distribution in this setting I would simply clip my actions in the environment. If, for example, the actor gives an action below your minimum value $X$, lets say it gives $X-0.5$, then I would simply clip the action to be $X$ when executed in the environment. This way your actor can still sample from a normal distribution which could give answers below $X$ (or above $Y$) and be used with your environment.
  
  If, for instance, your desired range was $(-1, 1)$ then another option would be to define your distribution to be $Y = \mbox{tanh}(X)$, where $X \sim N(\mu, \sigma)$. You can then find the density function of $Y$ using e.g. the [density transformation method](https://en.wikipedia.org/wiki/Probability_density_function#Function_of_random_variables_and_change_of_variables_in_the_probability_density_function).
  
  > **Comments:**
  > **username_0**: I see, thank you so much for clearing that about the normal distribution. From what I understand, the normal distribution is just a special case of the Gaussian distribution? Is that correct?  What if, for example, the output of the actor leads to an action range that is much smaller than the actual range of actions. That is, what if the range of output is [-2, 2] but the actual action range is [-5, 5]. Is expanding the output (unsquashing it) into the original action range of [-5,5] a valid thing to do?  
  > **username_1**: the normal distribution _is_ the Gaussian distribution, it is just a different name that people use :) in that instance, I would recommend to do some kind of normalisation so that the actors range equals the actual environments range. If the actual environment range is symmetric, i.e. $[-x, x]$ then you could use a `tanh` activation function for the final layer of the network and multiply the output by $x$.  
  > **username_0**: Yeah, sadly my action range isn't symmetric! But thanks for the help!  
  > **username_1**: you could try shifting you action range to start from 0, i.e. if the range is $[x, y]$ then make the range $[0, y-x]$ and then you just need to use a `relu` activation and clip at $y-x$, or use a `sigmoid` and multiply by $y-x$.  
  
  ---
  
  