Answers.Body: <p>The first equation (the one from the video) looks very wrong, for a few different reasons:</p>
  <ul>
  <li>It doesn't involve state-action values <span class="math-container">$Q(s, a)$</span>, but only something that looks like &quot;state values&quot; <span class="math-container">$Q(s)$</span>. Q-learning is about state-action values, so there should be both a state and an action as &quot;input&quot; of the <span class="math-container">$Q$</span> function.</li>
  <li>It is not clear what the <span class="math-container">$\max$</span> operator is maximising over.</li>
  <li>There should be an <span class="math-container">$s'$</span> rather than <span class="math-container">$s$</span> for the first <span class="math-container">$Q(\cdot$</span>) inside the <span class="math-container">$\max$</span>.</li>
  <li>A part of the equation on the right-hand side is simply missing, it ends very abruptly and is incomplete.</li>
  </ul>
  <hr />
  <p>The update rule that you think is correct, is indeed correct. However, it can also be rewritten (will be mathematically equivalent) in a form that is more similar to the first one, albeit with all of the mistakes that I pointed out above fixed. I'll start with the equation that you already recognised as being correct, and in a few small steps re-write it in a different form:</p>
  <p><span class="math-container">\begin{align*}
  Q(s, a) &amp;\gets Q(s, a) + \alpha \left( R(s, a) + \gamma \max_{a'} Q(s', a') - Q(s, a) \right) \\
  %
  Q(s, a) &amp;\gets Q(s, a) + \alpha \left( R(s, a) + \gamma \max_{a'} Q(s', a') \right) - \alpha Q(s, a) \\
  %
  Q(s, a) &amp;\gets Q(s, a) - \alpha Q(s, a) + \alpha \left( R(s, a) + \gamma \max_{a'} Q(s', a') \right) \\
  %
  Q(s, a) &amp;\gets \left( 1 - \alpha \right) Q(s, a) + \alpha \left( R(s, a) + \gamma \max_{a'} Q(s', a') \right)
  \end{align*}</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-19T17:26:38.647Z
Answers.LastActivityDate: 2024-06-19T17:26:38.647Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>I found the following q-learning formula:</p>
  <p><a href="https://i.sstatic.net/4cTRQFLj.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/4cTRQFLj.png" alt="enter image description here" /></a></p>
  <p>in this youtube video: <a href="https://www.youtube.com/watch?v=4C133ilFm3Q&amp;t=521s" rel="nofollow noreferrer">https://www.youtube.com/watch?v=4C133ilFm3Q&amp;t=521s</a></p>
  <p>I'm now a bit confused, since I thought, that the following one is the correct one:
  <a href="https://i.sstatic.net/xFflBZ7i.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/xFflBZ7i.png" alt="enter image description here" /></a></p>
  <p>Could somebody explain me which one is the correct one?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-24T13:14:47.107Z
Comments.Text: Can you please replace the screenshots with actual formulas written in mathjax? You could just adapt the formulas written in the answer below.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><q-learning>
Title: Why are there two different q-learning formulas?
UserDisplayName: Hans123
ThreadText: # Why are there two different q-learning formulas?
  
  (Asked by: username_0 on 2024-06-19)
  
  I found the following q-learning formula:
  
  [![enter image description here](https://i.sstatic.net/4cTRQFLj.png)](https://i.sstatic.net/4cTRQFLj.png)
  
  in this youtube video: <https://www.youtube.com/watch?v=4C133ilFm3Q&t=521s>
  
  I'm now a bit confused, since I thought, that the following one is the correct one:
  [![enter image description here](https://i.sstatic.net/xFflBZ7i.png)](https://i.sstatic.net/xFflBZ7i.png)
  
  Could somebody explain me which one is the correct one?
  
  **Tags:** reinforcement-learning, q-learning
  
  > **Comments:**
  > **username_1**: Can you please replace the screenshots with actual formulas written in mathjax? You could just adapt the formulas written in the answer below.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-06-19. Score: 5 | Accepted answer
  
  The first equation (the one from the video) looks very wrong, for a few different reasons:
  
  * It doesn't involve state-action values $Q(s, a)$, but only something that looks like "state values" $Q(s)$. Q-learning is about state-action values, so there should be both a state and an action as "input" of the $Q$ function.
  * It is not clear what the $\max$ operator is maximising over.
  * There should be an $s'$ rather than $s$ for the first $Q(\cdot$) inside the $\max$.
  * A part of the equation on the right-hand side is simply missing, it ends very abruptly and is incomplete.
  
  ---
  
  The update rule that you think is correct, is indeed correct. However, it can also be rewritten (will be mathematically equivalent) in a form that is more similar to the first one, albeit with all of the mistakes that I pointed out above fixed. I'll start with the equation that you already recognised as being correct, and in a few small steps re-write it in a different form:
  
  \begin{align\*}
  Q(s, a) &\gets Q(s, a) + \alpha \left( R(s, a) + \gamma \max\_{a'} Q(s', a') - Q(s, a) \right) \\
  %
  Q(s, a) &\gets Q(s, a) + \alpha \left( R(s, a) + \gamma \max\_{a'} Q(s', a') \right) - \alpha Q(s, a) \\
  %
  Q(s, a) &\gets Q(s, a) - \alpha Q(s, a) + \alpha \left( R(s, a) + \gamma \max\_{a'} Q(s', a') \right) \\
  %
  Q(s, a) &\gets \left( 1 - \alpha \right) Q(s, a) + \alpha \left( R(s, a) + \gamma \max\_{a'} Q(s', a') \right)
  \end{align\*}
  
  
  ---
  
  