Answers.Body: <p>It is a reference to the difference between their update rules.</p>
  <p>While both are attempting to estimate <span class="math-container">$Q(S_t,A_t)$</span> by iterative updates, the formula they use to converge on this value is different.</p>
  <p>Expected SARSA uses a <strong>weighted sum</strong> over the latest estimations of <span class="math-container">$Q(s_{t+1}, a)$</span> for all actions in the new state (weights are the probability assigned to taking that action by the policy):</p>
  <p><span class="math-container">$$ Q(S_t,A_t) \leftarrow (1-\alpha)\cdot Q(S_t,A_t) + \alpha \cdot\left[R_{t+1} + \gamma\sum_a\pi(a | S_{t+1})Q(S_{t+1}, a)\right] $$</span></p>
  <p>Q-Learning uses the <strong>maximum</strong> expected value over all actions in the new state:</p>
  <p><span class="math-container">$$ Q(S_t,A_t) \leftarrow (1-\alpha)\cdot Q(S_t,A_t) + \alpha \cdot\left[R_{t+1} + \gamma\max_aQ(S_{t+1}, a)\right] $$</span></p>
  <p>So, to answer directly: the semi-circle represents taking the maximum.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-23T22:41:31.223Z
Answers.LastActivityDate: 2022-11-23T22:41:31.223Z
Answers.UserDisplayName: Multihunter
Body: <p>I might be blind.
  But I wasn't able to find or figure out what the small difference between Q-learn and SARSA depicts in the following image;
  <a href="https://i.sstatic.net/qeqwj.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/qeqwj.png" alt="enter image description here" /></a>
  (<a href="https://www.yanxurui.cc/posts/ai/2019-04-24-RL-Introduction/" rel="nofollow noreferrer">src</a>).
  What does the semi-circle show? and what does the lack of the semi-circle show? I've your eyes with some red arrows.</p>
  
Tags: <reinforcement-learning><q-learning><expected-sarsa>
Title: What does the figure in Q-learning vs Expected SARSA actually show?
UserDisplayName: nammerkage
ThreadText: # What does the figure in Q-learning vs Expected SARSA actually show?
  
  (Asked by: username_0 on 2022-11-23)
  
  I might be blind.
  But I wasn't able to find or figure out what the small difference between Q-learn and SARSA depicts in the following image;
  [![enter image description here](https://i.sstatic.net/qeqwj.png)](https://i.sstatic.net/qeqwj.png)
  ([src](https://www.yanxurui.cc/posts/ai/2019-04-24-RL-Introduction/)).
  What does the semi-circle show? and what does the lack of the semi-circle show? I've your eyes with some red arrows.
  
  **Tags:** reinforcement-learning, q-learning, expected-sarsa
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-11-23. Score: 2 | Accepted answer
  
  It is a reference to the difference between their update rules.
  
  While both are attempting to estimate $Q(S\_t,A\_t)$ by iterative updates, the formula they use to converge on this value is different.
  
  Expected SARSA uses a **weighted sum** over the latest estimations of $Q(s\_{t+1}, a)$ for all actions in the new state (weights are the probability assigned to taking that action by the policy):
  
  $$ Q(S\_t,A\_t) \leftarrow (1-\alpha)\cdot Q(S\_t,A\_t) + \alpha \cdot\left[R\_{t+1} + \gamma\sum\_a\pi(a | S\_{t+1})Q(S\_{t+1}, a)\right] $$
  
  Q-Learning uses the **maximum** expected value over all actions in the new state:
  
  $$ Q(S\_t,A\_t) \leftarrow (1-\alpha)\cdot Q(S\_t,A\_t) + \alpha \cdot\left[R\_{t+1} + \gamma\max\_aQ(S\_{t+1}, a)\right] $$
  
  So, to answer directly: the semi-circle represents taking the maximum.
  
  
  ---
  
  