Body: <p>I am working on a problem and want to explore if it can be solved with PPO (or other policy gradient methods). The problem is that the action space is a bit special, compared to classic RL environments.</p>
  <p>At each time <span class="math-container">$t$</span>, we chose between 4 actions: <span class="math-container">$a_1\in \{0, 1, 2, 3\}$</span>,
  but given <span class="math-container">$a_1 = 0 \text{ or } 2$</span>, we need to chose three more actions: <span class="math-container">$a_2, a_3, a_4$</span> (which all three can be chosen from categorical distributions).</p>
  <p>I know I can design this kind of policy myself and re-work the entropy terms, and so on for PPO.</p>
  <p>My question is: <em>is there any research into this kind of RL?</em></p>
  <p>I am having a hard time finding someone working with problems in which the actions chosen are dependent on other chosen at the same time. I have looked into Hierarchical RL, but the papers I have found have not worked with this particular kind of problem.</p>
  <p>If these action spaces were small (<span class="math-container">$a_2, a_3$</span> are chosen from categorical distributions with <span class="math-container">$\sim$</span>800 different options), one solution would be to roll it out into one big policy where each possible combination of actions is represented by one choice in the policy. But my concern of doing this with a bigger action space is that the choice of <span class="math-container">$a_1 = 1, 3$</span> where we don't choose the other separate actions will get lost in the policy.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-06T12:57:05.693Z
Comments.Text: you could check out the options framework (I think mainly introduced by Doima Precup in her PhD work) but it works somewhat similar to what you are describing; at time $t$ you choose an option which then dictates that you follow an options policy until some stopping criteria is met (its been a while since I looked at this so I may not be being 100% accurate, but the general idea should be right).
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-09T03:39:00.927Z
Comments.Text: I made changes to your post to improve the clarity. Make sure the meaning has not changed.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><reference-request><policy-gradients><proximal-policy-optimization><action-spaces>
Title: Is there any research on the application of policy gradients to problems where the selection of an action requires the selection of another one?
UserDisplayName: Nathanson
ThreadText: # Is there any research on the application of policy gradients to problems where the selection of an action requires the selection of another one?
  
  (Asked by: username_0 on 2021-04-06)
  
  I am working on a problem and want to explore if it can be solved with PPO (or other policy gradient methods). The problem is that the action space is a bit special, compared to classic RL environments.
  
  At each time $t$, we chose between 4 actions: $a\_1\in \{0, 1, 2, 3\}$,
  but given $a\_1 = 0 \text{ or } 2$, we need to chose three more actions: $a\_2, a\_3, a\_4$ (which all three can be chosen from categorical distributions).
  
  I know I can design this kind of policy myself and re-work the entropy terms, and so on for PPO.
  
  My question is: *is there any research into this kind of RL?*
  
  I am having a hard time finding someone working with problems in which the actions chosen are dependent on other chosen at the same time. I have looked into Hierarchical RL, but the papers I have found have not worked with this particular kind of problem.
  
  If these action spaces were small ($a\_2, a\_3$ are chosen from categorical distributions with $\sim$800 different options), one solution would be to roll it out into one big policy where each possible combination of actions is represented by one choice in the policy. But my concern of doing this with a bigger action space is that the choice of $a\_1 = 1, 3$ where we don't choose the other separate actions will get lost in the policy.
  
  **Tags:** reinforcement-learning, reference-request, policy-gradients, proximal-policy-optimization, action-spaces
  > **Comments:**
  > **username_1**: you could check out the options framework (I think mainly introduced by Doima Precup in her PhD work) but it works somewhat similar to what you are describing; at time $t$ you choose an option which then dictates that you follow an options policy until some stopping criteria is met (its been a while since I looked at this so I may not be being 100% accurate, but the general idea should be right).  
  > **username_2**: I made changes to your post to improve the clarity. Make sure the meaning has not changed.  
  
  ---
  
  ## 0 answers
  
  