Answers.Body: <blockquote>
  <p>Do I have to calculate the Q for every possible action in my action space</p>
  </blockquote>
  <p>Yes, for the given state vector <span class="math-container">$\phi_{j+1}$</span> you must calculate the action value for all possible actions from that state. That is the only way that you can calculate <span class="math-container">$\text{max}_{a'} Q(\phi_{j+1}, a'; \theta)$</span> used to set the TD target <span class="math-container">$y_j$</span>.</p>
  <blockquote>
  <p>or simply the Q of the state?</p>
  </blockquote>
  <p>That is not defined. Q is an action value, and can only be calculated when you know both the state and the action.</p>
  <p>In practice, your neural network in DQN will be one of two designs:</p>
  <ul>
  <li><p>It takes state and action concatenated as input and models the state-action value function directly: <span class="math-container">$Q(s,a): \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$</span>. In this case you will need to run a mini-batch of all allowed actions in the same state in order to find <span class="math-container">$\text{max}_{a'} Q(\phi_{j+1}, a'; \theta)$</span></p>
  </li>
  <li><p>It takes state as input and models the state-action value for all actions at once: <span class="math-container">$f(s): \mathcal{S} \rightarrow \mathbb{R}^{|\mathcal{A}|}$</span>. In this case you will only need to run the forward action once to find <span class="math-container">$\text{max}_{a'} Q(\phi_{j+1}, a'; \theta)$</span>, but you will need to manipulate the resulting TD target data to ensure the only error gradient is from the selected action - typically this is done by running the network forward on the current state <span class="math-container">$\phi_{j}$</span> and modifying that vector with the new TD taget only for the taken action <span class="math-container">$a_j$</span></p>
  </li>
  </ul>
  <p>These designs are not mentioned in the pseudo-code because they are an implementation detail somewhat separate from the theory. You will find the second design is more common in DQN implementations, as it often has better performance (in terms of raw speed processing each state).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-14T12:30:28.923Z
Answers.Text: Thank you. Truth be told, my implementation is V-value rather than Q-value for various reasons, thus the confusion on my part.
Answers.UserDisplayName: Antonis Karvelas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-14T12:37:37.883Z
Answers.Text: @AntonisKarvelas You can use state value V if you are using a model and/or *afterstates* (which are a kind of hybrid state/action and common in e.g. board and card games)- in which case you will need to code to look ahead to get all possible $V(s')$ depending on the action or afterstate choices. You will almost definitely be using the first architecture in your case.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-14T12:39:56.390Z
Answers.Text: I'm doing exactly this. Is there a paper or an implementation I can look into? So far I'm following my idea of this hybrid and I may be doing something silly.
Answers.UserDisplayName: Antonis Karvelas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-14T12:46:27.067Z
Answers.Text: @AntonisKarvelas I am sure there will be some implentations using DQN online, as well as various MCTS/"Alpha Zero" variations. For just the Q-learning side I produced a tic tac toe learner in Python and published in on Kaggle here: https://www.kaggle.com/code/slobo777/tic-tac-toe-agent-using-q-learning (the original repo I deleted, as I never filled in with planned other games)
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-14T10:08:09.317Z
Answers.LastActivityDate: 2022-04-14T10:34:51.113Z
Answers.LastEditDate: 2022-04-14T10:34:51.113Z
Answers.UserDisplayName: Neil Slater
Body: <p>I'm trying to implement the DQN paper using python/pytorch for my needs (<a href="https://www.cs.toronto.edu/%7Evmnih/docs/dqn.pdf" rel="nofollow noreferrer">https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf</a>). I'm studying the main algorithm:
  <a href="https://i.sstatic.net/kFIrS.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/kFIrS.png" alt="enter image description here" /></a></p>
  <p>I am a bit confused about the <span class="math-container">$\gamma* \max Q$</span> when setting the <span class="math-container">$y$</span>. My model essentially takes a state and outputs a single float value (reward, cost, call it whatever). Do I have to calculate the Q for every possible action in my action space for that particular state or simply the Q of the state?</p>
  
Tags: <reinforcement-learning><deep-rl><q-learning><papers><dqn>
Title: When calculating the max in DQN, do I have to calculate the Q for every possible action for a particular state?
UserDisplayName: Antonis Karvelas
ThreadText: # When calculating the max in DQN, do I have to calculate the Q for every possible action for a particular state?
  
  (Asked by: username_0 on 2022-04-14)
  
  I'm trying to implement the DQN paper using python/pytorch for my needs ([https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf](https://www.cs.toronto.edu/%7Evmnih/docs/dqn.pdf)). I'm studying the main algorithm:
  [![enter image description here](https://i.sstatic.net/kFIrS.png)](https://i.sstatic.net/kFIrS.png)
  
  I am a bit confused about the $\gamma\* \max Q$ when setting the $y$. My model essentially takes a state and outputs a single float value (reward, cost, call it whatever). Do I have to calculate the Q for every possible action in my action space for that particular state or simply the Q of the state?
  
  **Tags:** reinforcement-learning, deep-rl, q-learning, papers, dqn
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-04-14. Score: 1 | Accepted answer
  
  > Do I have to calculate the Q for every possible action in my action space
  
  Yes, for the given state vector $\phi\_{j+1}$ you must calculate the action value for all possible actions from that state. That is the only way that you can calculate $\text{max}\_{a'} Q(\phi\_{j+1}, a'; \theta)$ used to set the TD target $y\_j$.
  
  > or simply the Q of the state?
  
  That is not defined. Q is an action value, and can only be calculated when you know both the state and the action.
  
  In practice, your neural network in DQN will be one of two designs:
  
  * It takes state and action concatenated as input and models the state-action value function directly: $Q(s,a): \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$. In this case you will need to run a mini-batch of all allowed actions in the same state in order to find $\text{max}\_{a'} Q(\phi\_{j+1}, a'; \theta)$
  * It takes state as input and models the state-action value for all actions at once: $f(s): \mathcal{S} \rightarrow \mathbb{R}^{|\mathcal{A}|}$. In this case you will only need to run the forward action once to find $\text{max}\_{a'} Q(\phi\_{j+1}, a'; \theta)$, but you will need to manipulate the resulting TD target data to ensure the only error gradient is from the selected action - typically this is done by running the network forward on the current state $\phi\_{j}$ and modifying that vector with the new TD taget only for the taken action $a\_j$
  
  These designs are not mentioned in the pseudo-code because they are an implementation detail somewhat separate from the theory. You will find the second design is more common in DQN implementations, as it often has better performance (in terms of raw speed processing each state).
  
  > **Comments:**
  > **username_0**: Thank you. Truth be told, my implementation is V-value rather than Q-value for various reasons, thus the confusion on my part.  
  > **username_1**: @AntonisKarvelas You can use state value V if you are using a model and/or *afterstates* (which are a kind of hybrid state/action and common in e.g. board and card games)- in which case you will need to code to look ahead to get all possible $V(s')$ depending on the action or afterstate choices. You will almost definitely be using the first architecture in your case.  
  > **username_0**: I'm doing exactly this. Is there a paper or an implementation I can look into? So far I'm following my idea of this hybrid and I may be doing something silly.  
  > **username_1**: @AntonisKarvelas I am sure there will be some implentations using DQN online, as well as various MCTS/"Alpha Zero" variations. For just the Q-learning side I produced a tic tac toe learner in Python and published in on Kaggle here: https://www.kaggle.com/code/slobo777/tic-tac-toe-agent-using-q-learning (the original repo I deleted, as I never filled in with planned other games)  
  
  ---
  
  