Answers.Body: <p>A side note for other people who have similar questions. A large number of actions is also problematic for the policy network. Basically, the policy network has an output for every action (one-hot encoding), and computing softmax over a large number of classes (more than 1K approximately) is not viable. One can use other activation functions or techniques that make a network more generalizable for such a large number of classes.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-08T09:48:09.653Z
Answers.LastActivityDate: 2023-07-08T09:48:09.653Z
Answers.UserDisplayName: abolfazl taghribi
Answers.Body: <p>I don't think that (at least from a practical standpoint), there is much difference between continuous action space and discrete action space with &gt;2k discrete actions. Quoting the <a href="https://arxiv.org/abs/1509.02971" rel="noreferrer"><em>&quot;Continuous control with Deep RL&quot;</em></a> paper - which I'd recommend as a starting point for your investigation:</p>
  <blockquote>
  <p>An obvious approach to adapting deep reinforcement learning methods such as DQN to continuous
  domains is to to simply discretize the action space. ... Such large action spaces are difficult to explore efficiently, and thus successfully training
  DQN-like networks in this context is likely intractable. Additionally, naive discretization of action spaces needlessly throws away information about the structure of the action domain, which may be essential for solving many problems.</p>
  </blockquote>
  <p>The last sentence in the quote above is the most important point for dealing with your problem. The fundamental issue is inability to efficiently explore such a large  action space - so the idea is to use its structure. I'm sure that your &gt;2k discrete action set has a certain structure on it. Like some actions might be &quot;closer&quot; to others. If so, then you (1) can infer some information about &quot;neighbor&quot; actions even if you never took them (2) do some exploration by adding noise to your policy-preferred action.</p>
  <p>The Actor-Critic class of algorithms matches perfectly the steps (1) and (2) above. The Critic Q-value network learns about your state-action space and the Actor policy network returns actions that you could smear.</p>
  <blockquote>
  <p>Are policy gradient methods good for large discrete action spaces?</p>
  </blockquote>
  <p>The Actor-Critic class of RL algorithms is a subclass of the Policy Gradient algorithms. So, the answer is yes - it looks like going that way is your best shot at making progress.</p>
  <blockquote>
  <p>Does this effectively mean DQN will <strong>not</strong> work?</p>
  </blockquote>
  <p>I don't think that there is a strict &quot;it will <strong>not</strong> work&quot; statement. The problem is that it just would be extremely difficult to make the DQN training stable &quot;by hand&quot;. <em>In principle</em> you might be able to construct a neural architecture that captures the state-action space structure. And then figure out how to perform efficient exploration. And then ensure that nothing explodes anywhere. But that's exactly what DDPG and then TRPO/PPO approaches would do for you.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-18T17:52:15.820Z
Answers.Text: thank you! this is exactly what i was looking for - it looks like actor critic is the way i will go (i will read this paper properly you linked properly.. I am just starting to learning about policy gradients) - although having said that - given the relative simplicity of the task (as David pointed out above) - i will also use DQN to see how it goes.. The exploration point is a good one, i was thinking about reward based exploration
  
  thank you for your time
Answers.UserDisplayName: user9317212
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-18T17:28:31.477Z
Answers.LastActivityDate: 2021-05-18T17:28:31.477Z
Answers.UserDisplayName: Kostya
Body: <p>I have seen this question asked primarily in the context of continuous action spaces.</p>
  <p>I have a large action space (~2-4k discrete actions) for my custom environment that I cannot reduce down further:</p>
  <p>I am currently trying DQN approaches but was wondering that given the large action space - if policy gradient methods are more appropriate <strong>and</strong> if they are appropriate for large action spaces that are discrete as in my scenario above. I have seen answers to this question with regard to large continuous action spaces.</p>
  <p>Finally - I imagine there isn't a simple answer to this but: Does this effectively mean DQN will <strong>not</strong> work?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-18T15:37:47.260Z
Comments.Text: I don't think there's a definitive answer to this question. Obviously having a large discrete action space would make learning harder, but it would also depend on the complexity of the task. [This](https://arxiv.org/pdf/1512.07679) is a good paper that looks at learning with large discrete action spaces.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-18T17:52:58.207Z
Comments.Text: thank you! I will try a DQN as well as a policy gradient method as the task itself is relatively simple
Comments.UserDisplayName: user9317212
Tags: <reinforcement-learning><dqn><policy-gradients>
Title: Are policy gradient methods good for large discrete action spaces?
UserDisplayName: user9317212
ThreadText: # Are policy gradient methods good for large discrete action spaces?
  
  (Asked by: username_0 on 2021-05-18)
  
  I have seen this question asked primarily in the context of continuous action spaces.
  
  I have a large action space (~2-4k discrete actions) for my custom environment that I cannot reduce down further:
  
  I am currently trying DQN approaches but was wondering that given the large action space - if policy gradient methods are more appropriate **and** if they are appropriate for large action spaces that are discrete as in my scenario above. I have seen answers to this question with regard to large continuous action spaces.
  
  Finally - I imagine there isn't a simple answer to this but: Does this effectively mean DQN will **not** work?
  
  **Tags:** reinforcement-learning, dqn, policy-gradients
  > **Comments:**
  > **username_1**: I don't think there's a definitive answer to this question. Obviously having a large discrete action space would make learning harder, but it would also depend on the complexity of the task. [This](https://arxiv.org/pdf/1512.07679) is a good paper that looks at learning with large discrete action spaces.  
  > **username_0**: thank you! I will try a DQN as well as a policy gradient method as the task itself is relatively simple  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2021-05-18. Score: 8 | Accepted answer
  
  I don't think that (at least from a practical standpoint), there is much difference between continuous action space and discrete action space with >2k discrete actions. Quoting the [*"Continuous control with Deep RL"*](https://arxiv.org/abs/1509.02971) paper - which I'd recommend as a starting point for your investigation:
  
  > An obvious approach to adapting deep reinforcement learning methods such as DQN to continuous
  > domains is to to simply discretize the action space. ... Such large action spaces are difficult to explore efficiently, and thus successfully training
  > DQN-like networks in this context is likely intractable. Additionally, naive discretization of action spaces needlessly throws away information about the structure of the action domain, which may be essential for solving many problems.
  
  The last sentence in the quote above is the most important point for dealing with your problem. The fundamental issue is inability to efficiently explore such a large action space - so the idea is to use its structure. I'm sure that your >2k discrete action set has a certain structure on it. Like some actions might be "closer" to others. If so, then you (1) can infer some information about "neighbor" actions even if you never took them (2) do some exploration by adding noise to your policy-preferred action.
  
  The Actor-Critic class of algorithms matches perfectly the steps (1) and (2) above. The Critic Q-value network learns about your state-action space and the Actor policy network returns actions that you could smear.
  
  > Are policy gradient methods good for large discrete action spaces?
  
  The Actor-Critic class of RL algorithms is a subclass of the Policy Gradient algorithms. So, the answer is yes - it looks like going that way is your best shot at making progress.
  
  > Does this effectively mean DQN will **not** work?
  
  I don't think that there is a strict "it will **not** work" statement. The problem is that it just would be extremely difficult to make the DQN training stable "by hand". *In principle* you might be able to construct a neural architecture that captures the state-action space structure. And then figure out how to perform efficient exploration. And then ensure that nothing explodes anywhere. But that's exactly what DDPG and then TRPO/PPO approaches would do for you.
  
  > **Comments:**
  > **username_0**: thank you! this is exactly what i was looking for - it looks like actor critic is the way i will go (i will read this paper properly you linked properly.. I am just starting to learning about policy gradients) - although having said that - given the relative simplicity of the task (as David pointed out above) - i will also use DQN to see how it goes.. The exploration point is a good one, i was thinking about reward based exploration  thank you for your time  
  
  ---
  
  ### Answer by username_2 on 2023-07-08. Score: -1
  
  A side note for other people who have similar questions. A large number of actions is also problematic for the policy network. Basically, the policy network has an output for every action (one-hot encoding), and computing softmax over a large number of classes (more than 1K approximately) is not viable. One can use other activation functions or techniques that make a network more generalizable for such a large number of classes.
  
  
  ---
  
  