Answers.Body: <p>What you describe will be possible, but you need to ensure that your policy returns the probability of the action(s) taken.</p>
  <p>In the two step process you describe, you would have something like
  <span class="math-container">\begin{align*} 
  \mathbb{P}(a_t) = \mathbb{P}(a_{(t, 1)}=A)\mathbb{P}(a_{(t, 2)}=B \, | \, a_{(t, 1)}=A)
  \end{align*}</span>
  where <span class="math-container">$a_t$</span> is the action taken, which consists of the two steps <span class="math-container">$a_{(t,1)}$</span> and <span class="math-container">$a_{(t,2)}$</span>.</p>
  <p>If you aren't strong in probability, here is another way to think about it. You have 60 possible actions (20 possibilities for A, and for each A, 3 possibilities for B). For each of the 60 actions, the policy must return the probability of selecting that (A, B) action tuple. If you sum the probability over all 60 of the action pairs, it must sum to 1. If it does not sum to 1, the gradient will not be correct.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-12T02:12:38.987Z
Answers.LastActivityDate: 2022-06-12T02:12:38.987Z
Answers.UserDisplayName: Taw
Body: <p>I have a custom PPO implementation, and it works fine, but I need to add to it the ability to select 2 actions per turn, one different in nature from the other, one dependent on the other.</p>
  <p>Imagine that a turn a had 20 possible actions. Now, for each of these action of type A, I need to choose one from 3 possible actions B. They are not the same, and thus they can not be on different turns.</p>
  <p>What I tried was to flatten the 20x3 space to a 60. Then, action A is 60//3 and action B is 60%3. But this does not train well. Are there any good methods for this issue?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-09T20:03:13.113Z
Comments.Text: Hi @Antonis Karvelas! Could you explain what you mean by the algorithm not training well? Thanks!
Comments.UserDisplayName: DeepQZero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-09T20:05:27.640Z
Comments.Text: Nothing converges. I imagine that it can pick good action A and bad action B or vice versa but it can't figure out which is which...
Comments.UserDisplayName: Antonis Karvelas
Tags: <reinforcement-learning><proximal-policy-optimization><discrete-action-spaces>
Title: PPO: multiple discrete actions per step, one depends on the other
UserDisplayName: Antonis Karvelas
ThreadText: # PPO: multiple discrete actions per step, one depends on the other
  
  (Asked by: username_0 on 2022-06-09)
  
  I have a custom PPO implementation, and it works fine, but I need to add to it the ability to select 2 actions per turn, one different in nature from the other, one dependent on the other.
  
  Imagine that a turn a had 20 possible actions. Now, for each of these action of type A, I need to choose one from 3 possible actions B. They are not the same, and thus they can not be on different turns.
  
  What I tried was to flatten the 20x3 space to a 60. Then, action A is 60//3 and action B is 60%3. But this does not train well. Are there any good methods for this issue?
  
  **Tags:** reinforcement-learning, proximal-policy-optimization, discrete-action-spaces
  
  > **Comments:**
  > **username_1**: Hi @username_0! Could you explain what you mean by the algorithm not training well? Thanks!  
  > **username_0**: Nothing converges. I imagine that it can pick good action A and bad action B or vice versa but it can't figure out which is which...  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-06-12. Score: 1
  
  What you describe will be possible, but you need to ensure that your policy returns the probability of the action(s) taken.
  
  In the two step process you describe, you would have something like
  \begin{align\*}
  \mathbb{P}(a\_t) = \mathbb{P}(a\_{(t, 1)}=A)\mathbb{P}(a\_{(t, 2)}=B \, | \, a\_{(t, 1)}=A)
  \end{align\*}
  where $a\_t$ is the action taken, which consists of the two steps $a\_{(t,1)}$ and $a\_{(t,2)}$.
  
  If you aren't strong in probability, here is another way to think about it. You have 60 possible actions (20 possibilities for A, and for each A, 3 possibilities for B). For each of the 60 actions, the policy must return the probability of selecting that (A, B) action tuple. If you sum the probability over all 60 of the action pairs, it must sum to 1. If it does not sum to 1, the gradient will not be correct.
  
  
  ---
  
  