Answers.Body: <blockquote>
  <p>More precisely: is DQNN applicable only when we have high translational invariance in our input(s)?</p>
  </blockquote>
  <p>No, DQN is not restricted to images or other kinds of inputs with those properties, it can be used with pretty much any kind of inputs.</p>
  <p>The <strong>DQN algorithm</strong> should be viewed separately from the <strong>Neural Network Architecture</strong> though. DQN can be used with any kind of Neural Network architecture. Yes, the most commonly-used type of architecture used with DQN is probably architectures that start with a bunch of convolutional layers, and those kinds of architectures are best suited for image-based inputs. This is not a requirement though. If you have other kinds of features where convolutional layers don't make a lot of sense, you can, for example, simply start directly with some ReLU layers.</p>
  <p>Here are two examples of DQN being used for problems without image-based inputs from the OpenAI baselines repository:</p>
  <ul>
  <li><a href="https://github.com/openai/baselines/blob/master/baselines/deepq/experiments/train_mountaincar.py" rel="nofollow noreferrer">DQN + Mountain Car</a></li>
  <li><a href="https://github.com/openai/baselines/blob/master/baselines/deepq/experiments/train_cartpole.py" rel="nofollow noreferrer">DQN + Cartpole</a></li>
  </ul>
  <p>In both cases, they're using <code>deepq.models.mlp()</code> to construct a relatively straightforward Multi-Layered Perceptron architecture, without any convolutional layers. In <a href="https://github.com/openai/baselines/blob/master/baselines/deepq/experiments/run_atari.py" rel="nofollow noreferrer">this example for Atari games</a>, they do have image-based input and therefore also construct an architecture with convolutional layers using <code>deepq.models.cnn_to_mlp()</code>.</p>
  <hr />
  <p>Note that, if you have relatively straightforward features, it may often not be necessary to use Deep RL approaches; tabular RL approaches of RL with Linear Function Approximation may work just as well in cases where you already have good features. If you have image-based inputs, those kinds of approaches are <em>much</em> less likely to work well. The disadvantage of Deep RL approaches like DQN is that they tend to require <strong>much more experience / data</strong> than simpler approaches.</p>
  <p>So, generally, the interesting question isn't &quot;<em>can DQN (or another Deep RL approach) handle my inputs?</em>&quot;, because the answer is probably yes. The more important question would often be &quot;<em>do I have to use DQN (or another Deep RL approach?</em>&quot;. The answer to that question will almost always be <em>yes</em> if you have image-based inputs, but relatively often be <em>no</em> if you already have good features as inputs.</p>
  <p>Another class of problems where Deep RL is really popular nowadays is <em>continuous control</em> problems (e.g. robot simulators, MuJoCo, etc.). These don't have image-based inputs, but still generally require Deep RL (not DQN though; DQN doesn't handle continuous-valued outputs very well, it generates discrete outputs).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-13T17:19:21.933Z
Answers.Text: So is the DQN algorithm basically only about the Q action value-function and the loss function both based on the reward?
Answers.UserDisplayName: gvgramazio
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-13T17:39:04.577Z
Answers.Text: @gvgramazio Pretty much yeah. I'd roughly define DQN as **a neural network** (architecture not specified other than that it has one output node per action) **+ Experience Replay Buffer + the loss function from the DQN paper + the idea of a Target Network**
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-13T16:50:52.540Z
Answers.LastActivityDate: 2021-12-16T22:08:10.730Z
Answers.LastEditDate: 2021-12-16T22:08:10.730Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>More precisely, is the DQN applicable only when we have high translational invariance in our input(s)?</p>
  <hr />
  <p>Starting from the original paper on nature (<a href="https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf" rel="nofollow noreferrer">here</a> a version stored on googleapis) and after looking online for some other implementation and based on the fact that this NN starts with convolutional layers, I think that is based on the assumption that we feed the network with images, but I'm not so sure.</p>
  <p>In the case that DQNN can be used with other types of inputs, please feel free to include examples in your answer. Also, references will be appreciated.</p>
  
Tags: <reinforcement-learning><reference-request><deep-rl><dqn>
Title: Is the DQN only applicable with images as inputs?
UserDisplayName: gvgramazio
ThreadText: # Is the DQN only applicable with images as inputs?
  
  (Asked by: username_0 on 2018-07-13)
  
  More precisely, is the DQN applicable only when we have high translational invariance in our input(s)?
  
  ---
  
  Starting from the original paper on nature ([here](https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf) a version stored on googleapis) and after looking online for some other implementation and based on the fact that this NN starts with convolutional layers, I think that is based on the assumption that we feed the network with images, but I'm not so sure.
  
  In the case that DQNN can be used with other types of inputs, please feel free to include examples in your answer. Also, references will be appreciated.
  
  **Tags:** reinforcement-learning, reference-request, deep-rl, dqn
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-07-13. Score: 5 | Accepted answer
  
  > More precisely: is DQNN applicable only when we have high translational invariance in our input(s)?
  
  No, DQN is not restricted to images or other kinds of inputs with those properties, it can be used with pretty much any kind of inputs.
  
  The **DQN algorithm** should be viewed separately from the **Neural Network Architecture** though. DQN can be used with any kind of Neural Network architecture. Yes, the most commonly-used type of architecture used with DQN is probably architectures that start with a bunch of convolutional layers, and those kinds of architectures are best suited for image-based inputs. This is not a requirement though. If you have other kinds of features where convolutional layers don't make a lot of sense, you can, for example, simply start directly with some ReLU layers.
  
  Here are two examples of DQN being used for problems without image-based inputs from the OpenAI baselines repository:
  
  * [DQN + Mountain Car](https://github.com/openai/baselines/blob/master/baselines/deepq/experiments/train_mountaincar.py)
  * [DQN + Cartpole](https://github.com/openai/baselines/blob/master/baselines/deepq/experiments/train_cartpole.py)
  
  In both cases, they're using `deepq.models.mlp()` to construct a relatively straightforward Multi-Layered Perceptron architecture, without any convolutional layers. In [this example for Atari games](https://github.com/openai/baselines/blob/master/baselines/deepq/experiments/run_atari.py), they do have image-based input and therefore also construct an architecture with convolutional layers using `deepq.models.cnn_to_mlp()`.
  
  ---
  
  Note that, if you have relatively straightforward features, it may often not be necessary to use Deep RL approaches; tabular RL approaches of RL with Linear Function Approximation may work just as well in cases where you already have good features. If you have image-based inputs, those kinds of approaches are *much* less likely to work well. The disadvantage of Deep RL approaches like DQN is that they tend to require **much more experience / data** than simpler approaches.
  
  So, generally, the interesting question isn't "*can DQN (or another Deep RL approach) handle my inputs?*", because the answer is probably yes. The more important question would often be "*do I have to use DQN (or another Deep RL approach?*". The answer to that question will almost always be *yes* if you have image-based inputs, but relatively often be *no* if you already have good features as inputs.
  
  Another class of problems where Deep RL is really popular nowadays is *continuous control* problems (e.g. robot simulators, MuJoCo, etc.). These don't have image-based inputs, but still generally require Deep RL (not DQN though; DQN doesn't handle continuous-valued outputs very well, it generates discrete outputs).
  
  > **Comments:**
  > **username_0**: So is the DQN algorithm basically only about the Q action value-function and the loss function both based on the reward?  
  > **username_1**: @username_0 Pretty much yeah. I'd roughly define DQN as **a neural network** (architecture not specified other than that it has one output node per action) **+ Experience Replay Buffer + the loss function from the DQN paper + the idea of a Target Network**  
  
  ---
  
  