Answers.Body: <p>I don't currently have much practical experience with DQN, but I can partially answer this question also based on my theoretical knowledge and other info that I found.</p>
  <p>DQN is <em>typically</em> used for</p>
  <ul>
  <li><p><strong>discrete action spaces</strong> (although there have been attempts to apply it to continuous action spaces, such as <a href="http://proceedings.mlr.press/v48/gu16.pdf" rel="nofollow noreferrer">this one</a>)</p>
  </li>
  <li><p><strong>discrete</strong> and <strong>continuous state spaces</strong></p>
  </li>
  <li><p>problems where the <strong>optimal policy is <a href="https://ai.stackexchange.com/q/12274/2444">deterministic</a></strong> (an example where the optimal policy is <strong>not</strong> deterministic is <a href="https://en.wikipedia.org/wiki/Rock_paper_scissors" rel="nofollow noreferrer">rock-paper-scissors</a>)</p>
  </li>
  <li><p><strong>off-policy</strong> learning (Q-learning is an off-policy algorithm, but the point is that, if you have a problem/application where data can only be or has been gathered by a policy that is unrelated to the policy you want to estimate, then DQN is suitable, though there are other off-policy algorithms, such as DDPG)</p>
  </li>
  </ul>
  <p><a href="https://stable-baselines3.readthedocs.io/en/master/guide/rl_tips.html" rel="nofollow noreferrer">This guide</a> also states that DQN is slower to train, but more <em>sample efficient</em> than other approaches, due to the use of the experience replay buffer.</p>
  <p>Moreover, if you have a small state and action space, it is probably a good idea to just use tabular Q-learning (i.e. no function approximation), given that <a href="https://ai.stackexchange.com/q/11679/2444">it is guaranteed to converge to the optimal value function</a>.</p>
  <p>See also <a href="https://ai.stackexchange.com/q/12255/2444">this</a> and <a href="https://ai.stackexchange.com/q/6196/2444">this</a> questions and <a href="https://flyyufelix.github.io/2017/10/12/dqn-vs-pg.html" rel="nofollow noreferrer">this article</a> (which compares DQN with policy gradients).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-21T21:20:28.467Z
Answers.LastActivityDate: 2020-11-22T20:03:13.030Z
Answers.LastEditDate: 2020-11-22T20:03:13.030Z
Answers.UserDisplayName: nbro
Body: <p>I know this is a general question, but I'm just looking for intuition. What are the characteristics of problems (in terms of state-space, action-space, environment, or anything else you can think of) that are well solvable with the family of DQN algorithms? What kind of problems are not well fit for DQNs?</p>
  
Tags: <neural-networks><reinforcement-learning><dqn><applications>
Title: What kind of problems is DQN algorithm good and bad for?
UserDisplayName: mark mark
ThreadText: # What kind of problems is DQN algorithm good and bad for?
  
  (Asked by: username_0 on 2020-11-19)
  
  I know this is a general question, but I'm just looking for intuition. What are the characteristics of problems (in terms of state-space, action-space, environment, or anything else you can think of) that are well solvable with the family of DQN algorithms? What kind of problems are not well fit for DQNs?
  
  **Tags:** neural-networks, reinforcement-learning, dqn, applications
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-11-21. Score: 2 | Accepted answer
  
  I don't currently have much practical experience with DQN, but I can partially answer this question also based on my theoretical knowledge and other info that I found.
  
  DQN is *typically* used for
  
  * **discrete action spaces** (although there have been attempts to apply it to continuous action spaces, such as [this one](http://proceedings.mlr.press/v48/gu16.pdf))
  * **discrete** and **continuous state spaces**
  * problems where the **optimal policy is [deterministic](https://ai.stackexchange.com/q/12274/2444)** (an example where the optimal policy is **not** deterministic is [rock-paper-scissors](https://en.wikipedia.org/wiki/Rock_paper_scissors))
  * **off-policy** learning (Q-learning is an off-policy algorithm, but the point is that, if you have a problem/application where data can only be or has been gathered by a policy that is unrelated to the policy you want to estimate, then DQN is suitable, though there are other off-policy algorithms, such as DDPG)
  
  [This guide](https://stable-baselines3.readthedocs.io/en/master/guide/rl_tips.html) also states that DQN is slower to train, but more *sample efficient* than other approaches, due to the use of the experience replay buffer.
  
  Moreover, if you have a small state and action space, it is probably a good idea to just use tabular Q-learning (i.e. no function approximation), given that [it is guaranteed to converge to the optimal value function](https://ai.stackexchange.com/q/11679/2444).
  
  See also [this](https://ai.stackexchange.com/q/12255/2444) and [this](https://ai.stackexchange.com/q/6196/2444) questions and [this article](https://flyyufelix.github.io/2017/10/12/dqn-vs-pg.html) (which compares DQN with policy gradients).
  
  
  ---
  
  