Body: <p>I would like to have the following vector for example [0.2,0.6,0.3,0.4,0.8] end up looking like this after training [0,1,0,0,1].
  In other words , rather than choosing one action, I'm choosing more than one. The reason is that my model consists of two models in sequence, both which must be trained by PPO. The first model receives the input and outputs the above vector, which is then fed into the the second model which outputs one action per vector.</p>
  <p>I will be using pytorch and in particular torch.distributions package.</p>
  <p>My code:</p>
  <pre><code>        probs = policy(torch.tensor(state).unsqueeze(0).float())
          # sample an action from that set of probs
          sampler = Categorical(probs)
          action = sampler.sample()
  </code></pre>
  <p>I'm thinking that the part i need to change is the <code>sampler=Categorical(probs)</code></p>
  <p>what would i need to change Categorical to in order for it to work and what other considerations would i need to apply to my code if i want it to work?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-08-07T13:13:41.277Z
Comments.Text: Is this not just a conditional distribution? i.e. you want to learn $\pi(a_1, a_2|s) = \pi(a_1 | s) \times \pi(a_2 | a_1, s)$ ?
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-08-07T13:27:47.880Z
Comments.Text: I think you need to reshape `probs` to `(N, C)` where `N` is the number of actions and `C` the number of classes (in your case `C=2`)
Comments.UserDisplayName: Luca Anzalone
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-08-07T13:41:35.187Z
Comments.Text: The number of classes is variable, i want to output a multi class vector rather than one for a particular action as it currently does
Comments.UserDisplayName: Tofara Moyo
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-08-07T13:58:16.293Z
Comments.Text: Why do you differentiate between the number of actions and the number of classes?
Comments.UserDisplayName: Tofara Moyo
Tags: <python><pytorch><proximal-policy-optimization>
Title: PPO with multiple actions per action vector
UserDisplayName: Tofara Moyo
ThreadText: # PPO with multiple actions per action vector
  
  (Asked by: username_0 on 2024-08-07)
  
  I would like to have the following vector for example [0.2,0.6,0.3,0.4,0.8] end up looking like this after training [0,1,0,0,1].
  In other words , rather than choosing one action, I'm choosing more than one. The reason is that my model consists of two models in sequence, both which must be trained by PPO. The first model receives the input and outputs the above vector, which is then fed into the the second model which outputs one action per vector.
  
  I will be using pytorch and in particular torch.distributions package.
  
  My code:
  
  ```
          probs = policy(torch.tensor(state).unsqueeze(0).float())
          # sample an action from that set of probs
          sampler = Categorical(probs)
          action = sampler.sample()
  
  ```
  
  I'm thinking that the part i need to change is the `sampler=Categorical(probs)`
  
  what would i need to change Categorical to in order for it to work and what other considerations would i need to apply to my code if i want it to work?
  
  **Tags:** python, pytorch, proximal-policy-optimization
  
  > **Comments:**
  > **username_1**: Is this not just a conditional distribution? i.e. you want to learn $\pi(a_1, a_2|s) = \pi(a_1 | s) \times \pi(a_2 | a_1, s)$ ?  
  > **username_2**: I think you need to reshape `probs` to `(N, C)` where `N` is the number of actions and `C` the number of classes (in your case `C=2`)  
  > **username_0**: The number of classes is variable, i want to output a multi class vector rather than one for a particular action as it currently does  
  > **username_0**: Why do you differentiate between the number of actions and the number of classes?  
  
  ---
  
  ## 0 answers
  
  