Answers.Body: <p>I think this was just a "clever" design choice. You can actually design a neural network (NN), to represent your Q function, which receives as input the state and an action and outputs the corresponding Q value. However, to obtain <span class="math-container">$\max_aQ(s', a)$</span> (which is a term of the update rule of the <a href="https://ai.stackexchange.com/a/10560/2444">Q-learning algorithm</a>) you would need a "forward pass" of this network for each possible action from <span class="math-container">$s'$</span>. By having a NN that outputs the Q value for each possible action from a given <span class="math-container">$s'$</span>, you will just need one forward pass of the NN to obtain <span class="math-container">$\max_aQ(s', a)$</span>, that is, you pick the highest Q value among the outputs of your NN.</p>
  
  <p>In the paper <a href="https://arxiv.org/pdf/1708.05866.pdf" rel="nofollow noreferrer">A Brief Survey of Deep Reinforcement Learning</a> (by Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage and Anil Anthony Bharath), at page 7, section "Value functions" (and subsection "Function Approximation and the DQN"), it's written </p>
  
  <blockquote>
    <p>It was designed such that the final fully connected layer outputs <span class="math-container">$Q^\pi(s,\cdot)$</span> for all action values in a discrete set of actions — in this case, the various directions of the joystick and the fire button. This not only enables the best action, <span class="math-container">$\text{argmax}_a Q^\pi(s, a)$</span>, to be chosen after a single forward pass of the network, but also allows the network to more easily encode action-independent knowledge in the lower, convolutional layers.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-19T09:27:11.333Z
Answers.LastActivityDate: 2019-02-20T11:32:01.903Z
Answers.LastEditDate: 2019-02-20T11:32:01.903Z
Answers.UserDisplayName: nbro
Body: <p>I am learning Deep RL following this tutorial: <a href="https://medium.freecodecamp.org/an-introduction-to-deep-q-learning-lets-play-doom-54d02d8017d8" rel="nofollow noreferrer">https://medium.freecodecamp.org/an-introduction-to-deep-q-learning-lets-play-doom-54d02d8017d8</a></p>
  
  <p>I understand everything but one detail:</p>
  
  <p>This image shows the difference between a classic Q learning table and a DNN.
  It states that a Q table needs a state-action pair as input and outputs the corresponding Q value whereas a Deep Q network needs the state as feature input and outputs the Q value for each action that can be made in that state.</p>
  
  <p>But shouldn't the state AND the action together be the input to the network and the network just outputs a single Q value?</p>
  
  <p><a href="https://i.sstatic.net/A9ulk.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/A9ulk.png" alt="enter image description here"></a></p>
  
Tags: <reinforcement-learning><q-learning><dqn><deep-rl><action-spaces>
Title: Why does Deep Q Network outputs multiple Q values?
UserDisplayName: NMO
ThreadText: # Why does Deep Q Network outputs multiple Q values?
  
  (Asked by: username_0 on 2019-02-19)
  
  I am learning Deep RL following this tutorial: <https://medium.freecodecamp.org/an-introduction-to-deep-q-learning-lets-play-doom-54d02d8017d8>
  
  I understand everything but one detail:
  
  This image shows the difference between a classic Q learning table and a DNN.
  It states that a Q table needs a state-action pair as input and outputs the corresponding Q value whereas a Deep Q network needs the state as feature input and outputs the Q value for each action that can be made in that state.
  
  But shouldn't the state AND the action together be the input to the network and the network just outputs a single Q value?
  
  [![enter image description here](https://i.sstatic.net/A9ulk.png)](https://i.sstatic.net/A9ulk.png)
  
  **Tags:** reinforcement-learning, q-learning, dqn, deep-rl, action-spaces
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-02-19. Score: 5 | Accepted answer
  
  I think this was just a "clever" design choice. You can actually design a neural network (NN), to represent your Q function, which receives as input the state and an action and outputs the corresponding Q value. However, to obtain $\max\_aQ(s', a)$ (which is a term of the update rule of the [Q-learning algorithm](https://ai.stackexchange.com/a/10560/2444)) you would need a "forward pass" of this network for each possible action from $s'$. By having a NN that outputs the Q value for each possible action from a given $s'$, you will just need one forward pass of the NN to obtain $\max\_aQ(s', a)$, that is, you pick the highest Q value among the outputs of your NN.
  
  In the paper [A Brief Survey of Deep Reinforcement Learning](https://arxiv.org/pdf/1708.05866.pdf) (by Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage and Anil Anthony Bharath), at page 7, section "Value functions" (and subsection "Function Approximation and the DQN"), it's written
  
  > It was designed such that the final fully connected layer outputs $Q^\pi(s,\cdot)$ for all action values in a discrete set of actions — in this case, the various directions of the joystick and the fire button. This not only enables the best action, $\text{argmax}\_a Q^\pi(s, a)$, to be chosen after a single forward pass of the network, but also allows the network to more easily encode action-independent knowledge in the lower, convolutional layers.
  
  
  ---
  
  