Answers.Body: <p>As I understand it, you have a problem with a large action space - a vector of 10 integer variables. You also have a constraint on what valid actions should look like.</p>
  <p>Even with the action vector discretised to integer amounts, there are millions of possible actions. This is beyond anything you can reasonably solve with value-based methods such as Q-learning. The problem is deriving the policy from the action value estimates. To select a greedy action, you need to find the action which maximises <span class="math-container">$\hat{q}(s,a, \theta)$</span>, which in your case would mean either an insanely large output vector (covering all possible action combinations) or very large input batches to maximise over.</p>
  <p>So, DQN is not really available to you as a method, <em>before</em> considering the constraint. What can you use instead? Any policy-gradient method or actor-critic method should work. These are more fiddly to understand and implement than value-based methods, although there are plenty of references for PyTorch, for example <a href="https://medium.com/pytorch/pfrl-a-pytorch-based-deep-rl-library-22cd048cf49c" rel="nofollow noreferrer">the PFRL library</a> implements A3C, PPO, DDPG which would all be suitable as a start.</p>
  <p>What a policy gradient method does for your problem is allow you to define a policy function <span class="math-container">$\pi(s, \theta)$</span> which will either output a single action or the parameters for an action probability distribution that you can sample from. The latter is actually more common, to create a neural network that outputs the parameters of a probability distribution. This allows for exploration in an on-policy approach. DDPG (Deep Deterministic Policy Gradients) is an example of a method that uses a deterministic policy function, but there is still an action sampling stage because DDPG adds a noise function to the policy in order to explore.</p>
  <p>In your case, you could build a policy network that output a vector of 10 real values to repesent the means of the distribution, plus either 1 or 10 standard deviations if you are not using something like DDPG. Then the action choice could be sampled from the distribution that this defined.</p>
  <p>This would not solve your other problem - a constraint on the sum of elements of the vector. You also have an implied constraint of a minimum value for each element.</p>
  <p>For the constraint, I suggest you do not attempt to model it directly in the policy function, but instead define a fixed (no learnable parameters) mapping function from something that is easier to model in the neural network, to the constrained version. For instance, whatever action vector is output by sampling the neural network action, you could clip to minimum zero, then sum elements, divide by this sum and multiply by 15. Putting this function outside of the agent for training purposes - either part of the environment, or a &quot;helper&quot; - should make the maths and using the framework easier.</p>
  <p>Optimising the raw, unconstrained policy function does mean you will have multiple equivalent policies once transformed, so makes it a less efficient search. However, this is offset by not needing to figure out valid probability distributions to sample from within the constrained space, or the gradients associated with the constraints.</p>
  <p>Success of this approach will depend on a few details:</p>
  <ul>
  <li><p>Choice of distribution function for selecting actions</p>
  </li>
  <li><p>Choice of mapping function to apply constraints</p>
  </li>
  <li><p>Feasibility of exploring the state and action space sufficiently to find near optimal behaviour</p>
  </li>
  </ul>
  <p>You have some influence over the first two issues - if you have some sense of what &quot;good&quot; actions will be in the problem then you can try to ensure that the representation covers them well. For example if you expect that the vector should have multiple zeroes, then you could ensure the sampled values can go below zero easily and that you use a clipping function so that you are likely to get a few zeroes.</p>
  <p>The last issue is beyond your control. It is possible the problem is too hard to explore using RL. This may be the case when only very specific action values out of the many possible will give you good results. RL relies on getting some kind of reward signal to guide improvements. If there is a very large search space and only sparse rewards in specific circumstances, then the trial and error process may never find the optimal behaviours.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-12T18:29:58.783Z
Answers.Text: It's great! Thanks a lot, Neil. I have also read about action branching in DQN. I think both solutions would be helpful.
Answers.UserDisplayName: Star
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-13T15:55:48.343Z
Answers.Text: I wonder if dividing the time step by the number of possible actions could be a good idea. It seems to be realistic. In real world usually the sequence of the selected actions is also relevant.
Answers.UserDisplayName: Hermes Morales
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-13T16:02:57.813Z
Answers.Text: @HermesMorales That can work for some optimisation tasks, such as those where this action sequence is meaningful - even better if there is some immediate reward signal. So e.g. for a knapsack type problem. If, as the OP states, there is no meaning to partial actions, and reward *has* to be calculated after the combined action, then there _might_ still be a representation benefit that results in simpler faster code, but it could even be counter-productive to exploration.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-10T19:26:38.773Z
Answers.LastActivityDate: 2021-08-11T05:59:04.173Z
Answers.LastEditDate: 2021-08-11T05:59:04.173Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am working on a DQN project with Pytorch, where I should choose multiple discrete actions, each in a range, say, <code>(0, 15)</code>. I am wondering how I can model it, such that the sum of actions is 15. Does anyone know how to model that?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-07T07:20:07.390Z
Comments.Text: Is there any constraint on the vector size for the combined actions? Would it be meaningful to have the agent pick one action at a time?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-08T16:10:41.480Z
Comments.Text: The agent needs to take a vector of actions each time. The sum over the elements must be a specific constant.
Comments.UserDisplayName: Star
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-08T16:58:41.767Z
Comments.Text: Is there a maximum size to the vector? As written, the agent could pick a vector of size 1,000,000 with most elements zero, and it is not clear whether this would be meaningful or useful for your use case. Even with a vector size limit of 10 you would have a huge action space to search, so it would be nice to know what the limits are.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-10T16:16:19.267Z
Comments.Text: *Real* numbers? Your action space is huge! I hope that you are expecting similar state/action vectors to have similar utility in general? It's workable to have a few exceptions where value changes rapidly between similar inputs, but if results are more arbitrary then your agent will never be able to find a good policy.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-11T03:09:30.533Z
Comments.Text: Sorry for the typo. I meant the elements are integer numbers. Please assume that the action vector is of size 10 and each element can pick an integer number in range(0,15).
Comments.UserDisplayName: Star
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-11T05:55:41.857Z
Comments.Text: Integer numbers are slightly more awkward to model a distribution function and constraint function for, but it is still possible and my answer is still applicable. DQN cannot cope with the size of your action space.
Comments.UserDisplayName: Neil Slater
Tags: <reinforcement-learning><deep-learning><deep-rl><dqn><pytorch>
Title: Deep Q-Learning with multiple discrete actions
UserDisplayName: Star
ThreadText: # Deep Q-Learning with multiple discrete actions
  
  (Asked by: username_0 on 2021-08-06)
  
  I am working on a DQN project with Pytorch, where I should choose multiple discrete actions, each in a range, say, `(0, 15)`. I am wondering how I can model it, such that the sum of actions is 15. Does anyone know how to model that?
  
  **Tags:** reinforcement-learning, deep-learning, deep-rl, dqn, pytorch
  > **Comments:**
  > **username_1**: Is there any constraint on the vector size for the combined actions? Would it be meaningful to have the agent pick one action at a time?  
  > **username_0**: The agent needs to take a vector of actions each time. The sum over the elements must be a specific constant.  
  > **username_1**: Is there a maximum size to the vector? As written, the agent could pick a vector of size 1,000,000 with most elements zero, and it is not clear whether this would be meaningful or useful for your use case. Even with a vector size limit of 10 you would have a huge action space to search, so it would be nice to know what the limits are.  
  > **username_1**: *Real* numbers? Your action space is huge! I hope that you are expecting similar state/action vectors to have similar utility in general? It's workable to have a few exceptions where value changes rapidly between similar inputs, but if results are more arbitrary then your agent will never be able to find a good policy.  
  > **username_0**: Sorry for the typo. I meant the elements are integer numbers. Please assume that the action vector is of size 10 and each element can pick an integer number in range(0,15).  
  > **username_1**: Integer numbers are slightly more awkward to model a distribution function and constraint function for, but it is still possible and my answer is still applicable. DQN cannot cope with the size of your action space.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-08-10. Score: 0
  
  As I understand it, you have a problem with a large action space - a vector of 10 integer variables. You also have a constraint on what valid actions should look like.
  
  Even with the action vector discretised to integer amounts, there are millions of possible actions. This is beyond anything you can reasonably solve with value-based methods such as Q-learning. The problem is deriving the policy from the action value estimates. To select a greedy action, you need to find the action which maximises $\hat{q}(s,a, \theta)$, which in your case would mean either an insanely large output vector (covering all possible action combinations) or very large input batches to maximise over.
  
  So, DQN is not really available to you as a method, *before* considering the constraint. What can you use instead? Any policy-gradient method or actor-critic method should work. These are more fiddly to understand and implement than value-based methods, although there are plenty of references for PyTorch, for example [the PFRL library](https://medium.com/pytorch/pfrl-a-pytorch-based-deep-rl-library-22cd048cf49c) implements A3C, PPO, DDPG which would all be suitable as a start.
  
  What a policy gradient method does for your problem is allow you to define a policy function $\pi(s, \theta)$ which will either output a single action or the parameters for an action probability distribution that you can sample from. The latter is actually more common, to create a neural network that outputs the parameters of a probability distribution. This allows for exploration in an on-policy approach. DDPG (Deep Deterministic Policy Gradients) is an example of a method that uses a deterministic policy function, but there is still an action sampling stage because DDPG adds a noise function to the policy in order to explore.
  
  In your case, you could build a policy network that output a vector of 10 real values to repesent the means of the distribution, plus either 1 or 10 standard deviations if you are not using something like DDPG. Then the action choice could be sampled from the distribution that this defined.
  
  This would not solve your other problem - a constraint on the sum of elements of the vector. You also have an implied constraint of a minimum value for each element.
  
  For the constraint, I suggest you do not attempt to model it directly in the policy function, but instead define a fixed (no learnable parameters) mapping function from something that is easier to model in the neural network, to the constrained version. For instance, whatever action vector is output by sampling the neural network action, you could clip to minimum zero, then sum elements, divide by this sum and multiply by 15. Putting this function outside of the agent for training purposes - either part of the environment, or a "helper" - should make the maths and using the framework easier.
  
  Optimising the raw, unconstrained policy function does mean you will have multiple equivalent policies once transformed, so makes it a less efficient search. However, this is offset by not needing to figure out valid probability distributions to sample from within the constrained space, or the gradients associated with the constraints.
  
  Success of this approach will depend on a few details:
  
  * Choice of distribution function for selecting actions
  * Choice of mapping function to apply constraints
  * Feasibility of exploring the state and action space sufficiently to find near optimal behaviour
  
  You have some influence over the first two issues - if you have some sense of what "good" actions will be in the problem then you can try to ensure that the representation covers them well. For example if you expect that the vector should have multiple zeroes, then you could ensure the sampled values can go below zero easily and that you use a clipping function so that you are likely to get a few zeroes.
  
  The last issue is beyond your control. It is possible the problem is too hard to explore using RL. This may be the case when only very specific action values out of the many possible will give you good results. RL relies on getting some kind of reward signal to guide improvements. If there is a very large search space and only sparse rewards in specific circumstances, then the trial and error process may never find the optimal behaviours.
  
  > **Comments:**
  > **username_0**: It's great! Thanks a lot, Neil. I have also read about action branching in DQN. I think both solutions would be helpful.  
  > **username_2**: I wonder if dividing the time step by the number of possible actions could be a good idea. It seems to be realistic. In real world usually the sequence of the selected actions is also relevant.  
  > **username_1**: @HermesMorales That can work for some optimisation tasks, such as those where this action sequence is meaningful - even better if there is some immediate reward signal. So e.g. for a knapsack type problem. If, as the OP states, there is no meaning to partial actions, and reward *has* to be calculated after the combined action, then there _might_ still be a representation benefit that results in simpler faster code, but it could even be counter-productive to exploration.  
  
  ---
  
  