Answers.Body: <p>If I read correctly, your RL action space is a <a href="https://github.com/openai/gym/blob/master/gym/spaces/multi_discrete.py" rel="nofollow noreferrer">Multi-Discrete</a> one, where each action is independent of each other and can be used simultaneously (like controller or keyboard), which is supported by <a href="https://github.com/DLR-RM/stable-baselines3" rel="nofollow noreferrer">Stable-baselines3 PPO algorithm</a> (but not DQN by the way). Choosing an action space is called &quot;action space shaping&quot;, and it is crucial for the performance of a RL algorithm.</p>
  <p>Luckily, <a href="https://arxiv.org/abs/2004.00980" rel="nofollow noreferrer">this paper</a> studies exactly just that, although it just studies PPO - but PPO is probably the most popular RL algorithm out there. It benchmarks 11 games over a wide range of action space shaping - like in your case, whether it should be 10 independent actions, or 1000 one-hot actions. The conclusion is that using a Multi Discrete action is a better choice:</p>
  <blockquote>
  <p>Avoid turning multi-discrete actions into a single discrete action and limit the number of choices per discrete action.</p>
  </blockquote>
  <p>So in your case, it is better to go a Multi Discrete with 10 actions. I highly recommend reading it for more insights.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-14T04:43:35.063Z
Answers.LastActivityDate: 2023-01-14T04:43:35.063Z
Answers.UserDisplayName: Minh-Long Luu
Body: <p>I'm interested in trying out Q-learning to solve a problem where I already have a simulation of the environment that can run at about 100,000 fps or steps/sec. Its also continuous with no terminal states.</p>
  <p>The estimated state space should be no more than 100,000. Most state can take integer values from 0 to 200.</p>
  <p>As for the action space, I am unsure if it should be 10, or if the action space should be 1000 (2**10). Basically there are 10 possible individual actions, but the actions can be pressed in all sorts of combinations, like action 1, 2, 3, and 5 can be taken at the same time.</p>
  <p>In this case, can tabular method still work fine? If so, are there any advantages of using a neural network, like DQN or PPO?</p>
  
Tags: <reinforcement-learning><deep-rl><dqn><markov-decision-process><proximal-policy-optimization>
Title: Should DQN/PPO be used for state spaces that are not that large?
UserDisplayName: gameveloster
ThreadText: # Should DQN/PPO be used for state spaces that are not that large?
  
  (Asked by: username_0 on 2023-01-14)
  
  I'm interested in trying out Q-learning to solve a problem where I already have a simulation of the environment that can run at about 100,000 fps or steps/sec. Its also continuous with no terminal states.
  
  The estimated state space should be no more than 100,000. Most state can take integer values from 0 to 200.
  
  As for the action space, I am unsure if it should be 10, or if the action space should be 1000 (2\*\*10). Basically there are 10 possible individual actions, but the actions can be pressed in all sorts of combinations, like action 1, 2, 3, and 5 can be taken at the same time.
  
  In this case, can tabular method still work fine? If so, are there any advantages of using a neural network, like DQN or PPO?
  
  **Tags:** reinforcement-learning, deep-rl, dqn, markov-decision-process, proximal-policy-optimization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-01-14. Score: 3
  
  If I read correctly, your RL action space is a [Multi-Discrete](https://github.com/openai/gym/blob/master/gym/spaces/multi_discrete.py) one, where each action is independent of each other and can be used simultaneously (like controller or keyboard), which is supported by [Stable-baselines3 PPO algorithm](https://github.com/DLR-RM/stable-baselines3) (but not DQN by the way). Choosing an action space is called "action space shaping", and it is crucial for the performance of a RL algorithm.
  
  Luckily, [this paper](https://arxiv.org/abs/2004.00980) studies exactly just that, although it just studies PPO - but PPO is probably the most popular RL algorithm out there. It benchmarks 11 games over a wide range of action space shaping - like in your case, whether it should be 10 independent actions, or 1000 one-hot actions. The conclusion is that using a Multi Discrete action is a better choice:
  
  > Avoid turning multi-discrete actions into a single discrete action and limit the number of choices per discrete action.
  
  So in your case, it is better to go a Multi Discrete with 10 actions. I highly recommend reading it for more insights.
  
  
  ---
  
  