Answers.Body: <p>The discussion uses poor notation, there should be a time index. You obtain a list of tuples <span class="math-container">$(s_t, a_t, r_t, s_{t+1})$</span> and then, for every visit MC, you update</p>
  <p><span class="math-container">$$Q(s_t, a_t) = Q(s_t, a_t) + \alpha (G_t - Q(s_t, a_t))\;;$$</span></p>
  <p>where <span class="math-container">$G_t = \sum_{k=0}^\infty \gamma^k r_{t+k}$</span>, for each <span class="math-container">$t$</span> in the episode. You can see that the returns for each time step are calculated for time timestep onwards, and so are not necessarily the same across time steps.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-20T14:41:01.990Z
Answers.Text: Thank you @DavidlIreland. So, in the case that there is only one reward at the final step as in chess, will every state-action pair, sampled in this episode, have the same return? Or generally, in the case that there are succesive actions with no rewards, will they share the return value in order to update themselves?
Answers.UserDisplayName: Hermes Morales
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-21T10:30:47.283Z
Answers.Text: They would be the same if the discount factor used is 1, if not then they will still be different.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-02T01:43:39.183Z
Answers.LastActivityDate: 2021-01-02T12:34:33.243Z
Answers.LastEditDate: 2021-01-02T12:34:33.243Z
Answers.UserDisplayName: David
Body: <p>Referring to <a href="https://ai.stackexchange.com/q/15431/2444">this post</a>, in the following formula to update the state-action value</p>
  <p><span class="math-container">$$ Q(s,a) = Q(s,a) + \alpha (G − Q(s,a)),$$</span></p>
  <p>is the value of <span class="math-container">$G$</span> (the return) the same for every state-action <span class="math-container">$(s,a)$</span> pair?</p>
  <p>I am a little confused about this point, so I will thank any clarification.</p>
  
Tags: <reinforcement-learning><monte-carlo-methods><return>
Title: When updating the state-action value in the Monte Carlo method, is the return the same for each state-action pair?
UserDisplayName: Hermes Morales
ThreadText: # When updating the state-action value in the Monte Carlo method, is the return the same for each state-action pair?
  
  (Asked by: username_0 on 2021-01-01)
  
  Referring to [this post](https://ai.stackexchange.com/q/15431/2444), in the following formula to update the state-action value
  
  $$ Q(s,a) = Q(s,a) + \alpha (G − Q(s,a)),$$
  
  is the value of $G$ (the return) the same for every state-action $(s,a)$ pair?
  
  I am a little confused about this point, so I will thank any clarification.
  
  **Tags:** reinforcement-learning, monte-carlo-methods, return
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-01-02. Score: 2 | Accepted answer
  
  The discussion uses poor notation, there should be a time index. You obtain a list of tuples $(s\_t, a\_t, r\_t, s\_{t+1})$ and then, for every visit MC, you update
  
  $$Q(s\_t, a\_t) = Q(s\_t, a\_t) + \alpha (G\_t - Q(s\_t, a\_t))\;;$$
  
  where $G\_t = \sum\_{k=0}^\infty \gamma^k r\_{t+k}$, for each $t$ in the episode. You can see that the returns for each time step are calculated for time timestep onwards, and so are not necessarily the same across time steps.
  
  > **Comments:**
  > **username_0**: Thank you @DavidlIreland. So, in the case that there is only one reward at the final step as in chess, will every state-action pair, sampled in this episode, have the same return? Or generally, in the case that there are succesive actions with no rewards, will they share the return value in order to update themselves?  
  > **username_1**: They would be the same if the discount factor used is 1, if not then they will still be different.  
  
  ---
  
  