Answers.Body: <p>Here is a table that attempts to <em>systematically</em> show the differences between tabular Q-learning (TQL), deep Q-learning (DQL), and deep Q-network (DQN).</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th></th>
  <th>Tabular Q-learning (TQL)</th>
  <th>Deep Q-learning (DQL)</th>
  <th>Deep Q-network (DQN)</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>Is it an RL algorithm?</td>
  <td>Yes</td>
  <td>Yes</td>
  <td>No (unless you use DQN to refer to DQL, which is done often!)</td>
  </tr>
  <tr>
  <td>Does it use neural networks?</td>
  <td>No. It uses a table.</td>
  <td>Yes</td>
  <td>No. DQN is the neural network.</td>
  </tr>
  <tr>
  <td>Is it a model?</td>
  <td>No</td>
  <td>No</td>
  <td><a href="https://ai.stackexchange.com/a/25460/2444">Yes</a> (<a href="https://ai.stackexchange.com/q/4456/2444">but usually not in the RL sense</a>)</td>
  </tr>
  <tr>
  <td>Can it deal with continuous state spaces?</td>
  <td>No (unless you discretize them)</td>
  <td><a href="https://ai.stackexchange.com/a/12257/2444">Yes</a></td>
  <td>Yes (in the sense that it can get real-valued inputs for the states)</td>
  </tr>
  <tr>
  <td>Can it deal with continuous action spaces?</td>
  <td><a href="https://ai.stackexchange.com/a/12257/2444">Yes</a> (but maybe not a good idea)</td>
  <td><a href="https://ai.stackexchange.com/a/12257/2444">Yes</a> (but maybe not a good idea)</td>
  <td>Yes (but only the sense that it can produce real-valued outputs for actions).</td>
  </tr>
  <tr>
  <td>Does it converge?</td>
  <td><a href="https://ai.stackexchange.com/q/11679/2444">Yes</a></td>
  <td><a href="https://ai.stackexchange.com/q/11679/2444">Not necessarily</a></td>
  <td><a href="https://ai.stackexchange.com/q/11679/2444">Not necessarily</a></td>
  </tr>
  <tr>
  <td>Is it an online learning algorithm?</td>
  <td>Yes</td>
  <td>No, if you use <a href="https://ai.stackexchange.com/q/6579/2444">experience replay</a></td>
  <td><a href="https://ai.stackexchange.com/q/10839/2444">No, but it can be used in an online learning setting</a></td>
  </tr>
  </tbody>
  </table>
  </div>
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-22T14:59:10.523Z
Answers.Text: Note that, when I say that neural networks are not online learning algorithms, I'm viewing neural networks as models and not algorithms. However, given that a specific neural network (with specific weights) computes some function and functions are algorithms (see [lambda calculus](https://en.wikipedia.org/wiki/Lambda_calculus)), they could also be considered _algorithms_. In most cases, the best way to think of a neural network is as a _model_ (i.e. something that you can use to approximate the desired function), but, sometimes, it's also useful to think of a neural network as a function.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-22T12:45:53.817Z
Answers.LastActivityDate: 2021-01-22T21:50:28.390Z
Answers.LastEditDate: 2021-01-22T21:50:28.390Z
Answers.UserDisplayName: nbro
Answers.Body: <p>Q-learning
  Q-learning is a basic reinforcement learning algorithm. It uses a Q-table to store and update the value of each state-action pair. The algorithm updates the Q-values using the Bellman equation based on the reward received and the estimated value of the next state. It's model-free, meaning it doesn't require a model of the environment and learns from interactions.</p>
  <p>Deep Q-learning
  Deep Q-learning is an extension of Q-learning that uses a deep neural network (DNN) instead of a Q-table. The Q-table becomes impractical in environments with large or continuous state spaces because it would need to store a Q-value for every possible state-action pair. Deep Q-learning uses a DNN to approximate the Q-values, allowing it to handle more complex environments.</p>
  <p>Deep Q-network (DQN)
  Deep Q-network (DQN) refers to the specific neural network architecture used in Deep Q-learning to approximate the Q-values. The DQN receives the state (often processed by convolutional layers in the case of image inputs) and outputs Q-values for all possible actions in that state.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-24T11:22:07.563Z
Answers.LastActivityDate: 2024-05-24T11:22:07.563Z
Answers.UserDisplayName: Kristina_Poole
Answers.Body: <p>In Q-learning (and in general value based reinforcement learning) we are typically interested in learning a Q-function, <span class="math-container">$Q(s, a)$</span>. This is defined as
  <span class="math-container">$$Q(s, a) = \mathbb{E}_\pi\left[ G_t | S_t = s, A_t = a \right]\;.$$</span></p>
  <p>For tabular Q-learning, where you have a finite state and action space you can maintain a table lookup that maintains your current estimate of the Q-value. Note that in practice even the spaces being finite might not be enough to <em>not</em> use function approximation, if e.g. your state space contains a large number, say <span class="math-container">$10^{10000}$</span>, of states, then it might not be manageable to maintain a separate Q-function for each state-action pair</p>
  <p>When you have an infinite state space (and/or action space) then it becomes impossible to use a table, and so you need to use function approximation to generalise across states. This is typically done using a deep neural network due to their expressive power -- aka, a deep Q-network (DQN). As a technical aside, the Q-networks don't usually take state <em>and</em> action as input, but take in a representation of the state (e.g. a <span class="math-container">$d$</span>-dimensional vector, or an image) and output a real valued vector of size <span class="math-container">$|\mathcal{A}|$</span>, where <span class="math-container">$\mathcal{A}$</span> is the action space.</p>
  <p>Now, it seems in your question that you're confused as to why you use a model (the neural network) when Q-learning is, as you rightly say, model-free. The answer here is that when we talk about RL algorithms being model-free we are not talking about how the value-functions or policy are parameterised, we are actually talking about whether the algorithms use a model of the transition dynamics to help with their learning. That is, a model free algorithm doesn't use any knowledge about <span class="math-container">$p(s' | s, a)$</span> (other than implicitly through repeated interactions with the environment), whereas model-based methods look to use this transition function explicitly - it can be known exactly such as in Atari environments, or it can approximated/learnt - to perform planning with the dynamics, or generate artificial data to learn from, etc.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-22T11:00:19.720Z
Answers.Text: what is that q-network output vector? kinda softmax prob values of all actions?
Answers.UserDisplayName: Dan D.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-22T11:04:09.103Z
Answers.Text: no, just real valued numbers. If you look at the expectation, it is not a probability nor does it have any bounds, so the output of the network reflects this.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-22T10:25:58.830Z
Answers.LastActivityDate: 2024-05-23T13:45:38.910Z
Answers.LastEditDate: 2024-05-23T13:45:38.910Z
Answers.UserDisplayName: David
Body: <p><strong>Q-learning</strong> uses a table to store all state-action pairs. Q-learning is a model-free RL algorithm, so how could there be the one called <strong>Deep Q-learning</strong>, as <em>deep</em> means using DNN; or maybe the state-action table (Q-table) is still there but the DNN is only for input reception (e.g. turning images into vectors)?</p>
  <p>Deep Q-network seems to be only the DNN part of the Deep Q-learning program, and Q-network seems the short for Deep Q-network.</p>
  <p><strong>Q-learning</strong>, <strong>Deep Q-learning</strong>, and <strong>Deep Q-network</strong>, what are the differences? May be there a comparison table between these 3 terms?</p>
  
Tags: <reinforcement-learning><comparison><q-learning><dqn><deep-rl>
Title: What is the difference between Q-learning, Deep Q-learning and Deep Q-network?
UserDisplayName: Dan D.
ThreadText: # What is the difference between Q-learning, Deep Q-learning and Deep Q-network?
  
  (Asked by: username_0 on 2021-01-22)
  
  **Q-learning** uses a table to store all state-action pairs. Q-learning is a model-free RL algorithm, so how could there be the one called **Deep Q-learning**, as *deep* means using DNN; or maybe the state-action table (Q-table) is still there but the DNN is only for input reception (e.g. turning images into vectors)?
  
  Deep Q-network seems to be only the DNN part of the Deep Q-learning program, and Q-network seems the short for Deep Q-network.
  
  **Q-learning**, **Deep Q-learning**, and **Deep Q-network**, what are the differences? May be there a comparison table between these 3 terms?
  
  **Tags:** reinforcement-learning, comparison, q-learning, dqn, deep-rl
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2021-01-22. Score: 9 | Accepted answer
  
  In Q-learning (and in general value based reinforcement learning) we are typically interested in learning a Q-function, $Q(s, a)$. This is defined as
  $$Q(s, a) = \mathbb{E}\_\pi\left[ G\_t | S\_t = s, A\_t = a \right]\;.$$
  
  For tabular Q-learning, where you have a finite state and action space you can maintain a table lookup that maintains your current estimate of the Q-value. Note that in practice even the spaces being finite might not be enough to *not* use function approximation, if e.g. your state space contains a large number, say $10^{10000}$, of states, then it might not be manageable to maintain a separate Q-function for each state-action pair
  
  When you have an infinite state space (and/or action space) then it becomes impossible to use a table, and so you need to use function approximation to generalise across states. This is typically done using a deep neural network due to their expressive power -- aka, a deep Q-network (DQN). As a technical aside, the Q-networks don't usually take state *and* action as input, but take in a representation of the state (e.g. a $d$-dimensional vector, or an image) and output a real valued vector of size $|\mathcal{A}|$, where $\mathcal{A}$ is the action space.
  
  Now, it seems in your question that you're confused as to why you use a model (the neural network) when Q-learning is, as you rightly say, model-free. The answer here is that when we talk about RL algorithms being model-free we are not talking about how the value-functions or policy are parameterised, we are actually talking about whether the algorithms use a model of the transition dynamics to help with their learning. That is, a model free algorithm doesn't use any knowledge about $p(s' | s, a)$ (other than implicitly through repeated interactions with the environment), whereas model-based methods look to use this transition function explicitly - it can be known exactly such as in Atari environments, or it can approximated/learnt - to perform planning with the dynamics, or generate artificial data to learn from, etc.
  
  > **Comments:**
  > **username_0**: what is that q-network output vector? kinda softmax prob values of all actions?  
  > **username_3**: no, just real valued numbers. If you look at the expectation, it is not a probability nor does it have any bounds, so the output of the network reflects this.  
  
  ---
  
  ### Answer by username_1 on 2021-01-22. Score: 13
  
  Here is a table that attempts to *systematically* show the differences between tabular Q-learning (TQL), deep Q-learning (DQL), and deep Q-network (DQN).
  
  |  | Tabular Q-learning (TQL) | Deep Q-learning (DQL) | Deep Q-network (DQN) |
  | --- | --- | --- | --- |
  | Is it an RL algorithm? | Yes | Yes | No (unless you use DQN to refer to DQL, which is done often!) |
  | Does it use neural networks? | No. It uses a table. | Yes | No. DQN is the neural network. |
  | Is it a model? | No | No | [Yes](https://ai.stackexchange.com/a/25460/2444) ([but usually not in the RL sense](https://ai.stackexchange.com/q/4456/2444)) |
  | Can it deal with continuous state spaces? | No (unless you discretize them) | [Yes](https://ai.stackexchange.com/a/12257/2444) | Yes (in the sense that it can get real-valued inputs for the states) |
  | Can it deal with continuous action spaces? | [Yes](https://ai.stackexchange.com/a/12257/2444) (but maybe not a good idea) | [Yes](https://ai.stackexchange.com/a/12257/2444) (but maybe not a good idea) | Yes (but only the sense that it can produce real-valued outputs for actions). |
  | Does it converge? | [Yes](https://ai.stackexchange.com/q/11679/2444) | [Not necessarily](https://ai.stackexchange.com/q/11679/2444) | [Not necessarily](https://ai.stackexchange.com/q/11679/2444) |
  | Is it an online learning algorithm? | Yes | No, if you use [experience replay](https://ai.stackexchange.com/q/6579/2444) | [No, but it can be used in an online learning setting](https://ai.stackexchange.com/q/10839/2444) |
  
  > **Comments:**
  > **username_1**: Note that, when I say that neural networks are not online learning algorithms, I'm viewing neural networks as models and not algorithms. However, given that a specific neural network (with specific weights) computes some function and functions are algorithms (see [lambda calculus](https://en.wikipedia.org/wiki/Lambda_calculus)), they could also be considered _algorithms_. In most cases, the best way to think of a neural network is as a _model_ (i.e. something that you can use to approximate the desired function), but, sometimes, it's also useful to think of a neural network as a function.  
  
  ---
  
  ### Answer by username_2 on 2024-05-24. Score: 1
  
  Q-learning
  Q-learning is a basic reinforcement learning algorithm. It uses a Q-table to store and update the value of each state-action pair. The algorithm updates the Q-values using the Bellman equation based on the reward received and the estimated value of the next state. It's model-free, meaning it doesn't require a model of the environment and learns from interactions.
  
  Deep Q-learning
  Deep Q-learning is an extension of Q-learning that uses a deep neural network (DNN) instead of a Q-table. The Q-table becomes impractical in environments with large or continuous state spaces because it would need to store a Q-value for every possible state-action pair. Deep Q-learning uses a DNN to approximate the Q-values, allowing it to handle more complex environments.
  
  Deep Q-network (DQN)
  Deep Q-network (DQN) refers to the specific neural network architecture used in Deep Q-learning to approximate the Q-values. The DQN receives the state (often processed by convolutional layers in the case of image inputs) and outputs Q-values for all possible actions in that state.
  
  
  ---
  
  