Answers.Body: <blockquote>
  <p><span class="math-container">$r_j + \gamma \max_{a'}Q(\phi_{j+1},a';\theta)$</span><br />
  I'm confused as to what <span class="math-container">$a'$</span> refers to and where it comes from.</p>
  </blockquote>
  <p>Here <span class="math-container">$a'$</span> is a &quot;dummy&quot; argument over which you perform the maximization operation <span class="math-container">$\max_{a'}$</span>.</p>
  <p>In practice, that would correspond to <code>axis</code> (or <code>dim</code>) argument in <a href="https://numpy.org/doc/stable/reference/generated/numpy.ndarray.max.html" rel="nofollow noreferrer">numpy</a>/<a href="https://pytorch.org/docs/stable/generated/torch.max.html" rel="nofollow noreferrer">pytorch</a>/<a href="https://www.tensorflow.org/api_docs/python/tf/math/reduce_max" rel="nofollow noreferrer">tensorflow</a></p>
  <blockquote>
  <p><span class="math-container">$a_t = \max_a Q^*(\phi(s_t),a;\theta)$</span><br />
  Why on the line 7 it's <span class="math-container">$a$</span></p>
  </blockquote>
  <p>I'd say that in this case it is a sloppy math notation (or just typo) on the authors' part.<br />
  It should be <code>argmax</code>, not <code>max</code>.
  <span class="math-container">$$a_t = \arg \max_a Q^*(\phi(s_t),a;\theta)$$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-09T16:48:00.497Z
Answers.Text: So, it doesn't refer to an action a'?
Answers.UserDisplayName: Ness
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-09T17:07:21.993Z
Answers.Text: Could you check my edit please?
Answers.UserDisplayName: Ness
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-09T17:13:20.557Z
Answers.Text: @Ness: It is both a dummy argument, and refers to a *potential* action taken as $a_{t+1}$, but not necessarily the *actual* observed action taken during training. For Q-learning, you don't look at the next real action choice, but optimise based on the current maximising action choice, whilst other methods may want to use expected action on the behaviour policy or real observed action that the agent takes next
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-09T16:36:02.923Z
Answers.LastActivityDate: 2023-01-09T18:12:03.153Z
Answers.LastEditDate: 2023-01-09T18:12:03.153Z
Answers.UserDisplayName: Kostya
Body: <p>I was reading this article <a href="https://www.cs.toronto.edu/%7Evmnih/docs/dqn.pdf" rel="nofollow noreferrer">https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf</a> and in it there is an algorithm of deep q learning with experience replay as follows:</p>
  <p><a href="https://i.sstatic.net/iUoeK.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/iUoeK.png" alt="enter image description here" /></a></p>
  <p>On line 12, when the algorithm is setting the values for <code>y_j</code>, the second line says:</p>
  <p><a href="https://i.sstatic.net/lcMvc.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/lcMvc.png" alt="enter image description here" /></a></p>
  <p>I'm confused as to what <code>a'</code> refers to and where it comes from.</p>
  <p>(Edit) Why on this line (line 7) it's <code>a</code>:</p>
  <p><a href="https://i.sstatic.net/YTlHP.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/YTlHP.png" alt="enter image description here" /></a></p>
  <p>But on line 12 it's <code>a'</code> ?</p>
  <p>Can someone please explain it to me?</p>
  
Tags: <reinforcement-learning><deep-rl><dqn><notation><pseudocode>
Title: In the DQN paper, why do we have both $\max_{a'}$ and $\max_{a}$ in the pseudocode?
UserDisplayName: Ness
ThreadText: # In the DQN paper, why do we have both $\max_{a'}$ and $\max_{a}$ in the pseudocode?
  
  (Asked by: username_0 on 2023-01-09)
  
  I was reading this article [https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf](https://www.cs.toronto.edu/%7Evmnih/docs/dqn.pdf) and in it there is an algorithm of deep q learning with experience replay as follows:
  
  [![enter image description here](https://i.sstatic.net/iUoeK.png)](https://i.sstatic.net/iUoeK.png)
  
  On line 12, when the algorithm is setting the values for `y_j`, the second line says:
  
  [![enter image description here](https://i.sstatic.net/lcMvc.png)](https://i.sstatic.net/lcMvc.png)
  
  I'm confused as to what `a'` refers to and where it comes from.
  
  (Edit) Why on this line (line 7) it's `a`:
  
  [![enter image description here](https://i.sstatic.net/YTlHP.png)](https://i.sstatic.net/YTlHP.png)
  
  But on line 12 it's `a'` ?
  
  Can someone please explain it to me?
  
  **Tags:** reinforcement-learning, deep-rl, dqn, notation, pseudocode
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-01-09. Score: 2
  
  > $r\_j + \gamma \max\_{a'}Q(\phi\_{j+1},a';\theta)$  
  > I'm confused as to what $a'$ refers to and where it comes from.
  
  Here $a'$ is a "dummy" argument over which you perform the maximization operation $\max\_{a'}$.
  
  In practice, that would correspond to `axis` (or `dim`) argument in [numpy](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.max.html)/[pytorch](https://pytorch.org/docs/stable/generated/torch.max.html)/[tensorflow](https://www.tensorflow.org/api_docs/python/tf/math/reduce_max)
  
  > $a\_t = \max\_a Q^\*(\phi(s\_t),a;\theta)$  
  > Why on the line 7 it's $a$
  
  I'd say that in this case it is a sloppy math notation (or just typo) on the authors' part.  
  It should be `argmax`, not `max`.
  $$a\_t = \arg \max\_a Q^\*(\phi(s\_t),a;\theta)$$
  
  > **Comments:**
  > **username_0**: So, it doesn't refer to an action a'?  
  > **username_0**: Could you check my edit please?  
  > **username_2**: @username_0: It is both a dummy argument, and refers to a *potential* action taken as $a_{t+1}$, but not necessarily the *actual* observed action taken during training. For Q-learning, you don't look at the next real action choice, but optimise based on the current maximising action choice, whilst other methods may want to use expected action on the behaviour policy or real observed action that the agent takes next  
  
  ---
  
  