Answers.Body: <p>At first glance, I thought this was similar to "continuous-discrete" action selection (<a href="https://arxiv.org/pdf/1810.06394.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1810.06394.pdf</a>). However, I think your problem is different.</p>
  
  <p>I am assuming that each <span class="math-container">$a_i$</span> is continuous and that the action which interacts with your environment is the entire vector <span class="math-container">$a = (a_1,a_2,\dotso,a_n)$</span> and not an individual <span class="math-container">$a_i$</span>. Then you could treat it like a hierarchical problem. If you want <span class="math-container">$a_1 + a_2 &lt; 2$</span> for example, then you could sample <span class="math-container">$a_1 \sim U(0,2)$</span> and <span class="math-container">$a_2 | a_1 \sim U(0, 2-a_1)$</span> and have <span class="math-container">$p(a) = p(a_2 | a_1)p(a_1)$</span>. The specifics of how you do this depends more finely on how your problem is set up.</p>
  
  <p>Perhaps you can find similar ideas from the paper linked above. Also, other work in the robitics literature studies structured and hybrid action spaces.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-27T14:08:30.410Z
Answers.Text: Thanks for the answer! Only issue for your method here is $a_1$ and $a_2$ are asymmetrical (e.g. $a_1$ always has higher mean that $a_2$). I kinda would like to treat each entry $a_i$ in a equal manner. I would definitely take a look a the paper you provided. I think policy gradient method can handle hybrid action space as long as the underlying probability distribution is properly specified.
Answers.UserDisplayName: Ryan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-28T02:10:45.893Z
Answers.Text: This is very contingent on the structure of your problem. Why not naively set $a_i \sim U(0,\frac{1}{n})$? I can't think of another general and non-hierarchical approach to this.
  
  If you do not want a systematic bias, then perhaps you could randomly sample the order for the indicies as well.
Answers.UserDisplayName: Alex L
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-28T14:09:30.077Z
Answers.Text: Having $a_i$ being uniform(0,1\n) will impose a 1/n bound on them, in addition, they will be independent. Remember I would like to have dependent entries... I only have one feasible way of doing so which I mention in my post.
Answers.UserDisplayName: Ryan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-29T01:13:56.810Z
Answers.Text: Right, I should have added general, non-hierarchical and *tractable*. 
  
  Since you are using policy gradient, don't you need to parameterize the distribution anyway? Perhaps you can simplify the problem by assuming IID while using lagrange multipliers to constrain the parameters.
Answers.UserDisplayName: Alex L
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-27T05:57:40.013Z
Answers.LastActivityDate: 2020-02-28T02:16:39.723Z
Answers.LastEditDate: 2020-02-28T02:16:39.723Z
Answers.UserDisplayName: Alex L
Body: <p>I am working on a customized RL environment where each action is represented as a tuple <span class="math-container">$a = (a_1,a_2,\cdots,a_n)$</span> such that certain condition must be satisfied for entries of <span class="math-container">$a$</span> (for instance, <span class="math-container">$a_1+a_2+\cdots+a_n \leq \text{constant}$</span>). </p>
  
  <p>I am using the policy gradient method, but I am having some difficulty modeling the underlying probability distribution of actions.  Is there any work done in this direction?</p>
  
  <p>For the constraint <span class="math-container">$a_1+a_2+\cdots+a_n \leq \text{constant}$</span>, I was thinking about generating <span class="math-container">$n+1$</span> uniform random variables <span class="math-container">$U_1,U_2,\cdots,U_n, U$</span>, and set <span class="math-container">$a_i = \text{constant}\times U \times \frac{U_i}{\sum_{j=1}^n U_j}$</span>. Problem is that the joint density is a bit messy to calculate, which is needed to get the negative log likelihood. I am curious about how such issue is handled in practice. </p>
  
Tags: <reinforcement-learning><policy-gradients><markov-decision-process>
Title: How can I constraint the actions with dependent coordinates?
UserDisplayName: Ryan
ThreadText: # How can I constraint the actions with dependent coordinates?
  
  (Asked by: username_0 on 2020-02-26)
  
  I am working on a customized RL environment where each action is represented as a tuple $a = (a\_1,a\_2,\cdots,a\_n)$ such that certain condition must be satisfied for entries of $a$ (for instance, $a\_1+a\_2+\cdots+a\_n \leq \text{constant}$).
  
  I am using the policy gradient method, but I am having some difficulty modeling the underlying probability distribution of actions. Is there any work done in this direction?
  
  For the constraint $a\_1+a\_2+\cdots+a\_n \leq \text{constant}$, I was thinking about generating $n+1$ uniform random variables $U\_1,U\_2,\cdots,U\_n, U$, and set $a\_i = \text{constant}\times U \times \frac{U\_i}{\sum\_{j=1}^n U\_j}$. Problem is that the joint density is a bit messy to calculate, which is needed to get the negative log likelihood. I am curious about how such issue is handled in practice.
  
  **Tags:** reinforcement-learning, policy-gradients, markov-decision-process
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-02-27. Score: 1
  
  At first glance, I thought this was similar to "continuous-discrete" action selection (<https://arxiv.org/pdf/1810.06394.pdf>). However, I think your problem is different.
  
  I am assuming that each $a\_i$ is continuous and that the action which interacts with your environment is the entire vector $a = (a\_1,a\_2,\dotso,a\_n)$ and not an individual $a\_i$. Then you could treat it like a hierarchical problem. If you want $a\_1 + a\_2 < 2$ for example, then you could sample $a\_1 \sim U(0,2)$ and $a\_2 | a\_1 \sim U(0, 2-a\_1)$ and have $p(a) = p(a\_2 | a\_1)p(a\_1)$. The specifics of how you do this depends more finely on how your problem is set up.
  
  Perhaps you can find similar ideas from the paper linked above. Also, other work in the robitics literature studies structured and hybrid action spaces.
  
  > **Comments:**
  > **username_0**: Thanks for the answer! Only issue for your method here is $a_1$ and $a_2$ are asymmetrical (e.g. $a_1$ always has higher mean that $a_2$). I kinda would like to treat each entry $a_i$ in a equal manner. I would definitely take a look a the paper you provided. I think policy gradient method can handle hybrid action space as long as the underlying probability distribution is properly specified.  
  > **username_1**: This is very contingent on the structure of your problem. Why not naively set $a_i \sim U(0,\frac{1}{n})$? I can't think of another general and non-hierarchical approach to this.  If you do not want a systematic bias, then perhaps you could randomly sample the order for the indicies as well.  
  > **username_0**: Having $a_i$ being uniform(0,1\n) will impose a 1/n bound on them, in addition, they will be independent. Remember I would like to have dependent entries... I only have one feasible way of doing so which I mention in my post.  
  > **username_1**: Right, I should have added general, non-hierarchical and *tractable*.   Since you are using policy gradient, don't you need to parameterize the distribution anyway? Perhaps you can simplify the problem by assuming IID while using lagrange multipliers to constrain the parameters.  
  
  ---
  
  