Answers.Body: <blockquote>
  <p>Question: is <span class="math-container">$\bar{R}_t$</span>  average of all rewards  or  average of  rewards corresponding to <span class="math-container">$A_t$</span>?</p>
  </blockquote>
  <p>It is average of all rewards seen so far. Usually a rolling recent average, so it slowly adapts to expected results from the current policy. But a mean value is good too.</p>
  <p>It is important that the value does not depend on the action choice, otherwise the gradient calculation will not be correct. In fact you can use pretty much <em>any</em> baseline provided it follows this rule, including fixed value guesses. Some baselines work out better than others in practice.</p>
  <p>If you have a contextual bandit (or an RL problem using a policy gradient method) then a good baseline choice might be average reward seen from the current state (note this depends on the <em>policy</em>, but not directly on the current action choice which might be an exploratory action). This leads to the various algorithms that use <em>advantage</em> - the difference between Q(s,a) and V(s) - as the scaling to <span class="math-container">$\nabla \text{log}(\pi(a|s))$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-30T21:43:06.947Z
Answers.Text: @ Neil Slater, in the book it says "which can be computed incrementally as described
  in Section 2.4 (or Section 2.5 if the problem is nonstationary)". However, in sections 2.4 and 2.5, we calculate average of rewards for a given action, not the average of all rewards seen so far. That was the reason why I am confused.
Answers.UserDisplayName: user3489173
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-01T08:58:25.727Z
Answers.Text: @user3489173 I think you are expected to extract the averaging technique from that reference, not what was being averaged.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-25T18:09:31.003Z
Answers.Text: @NeilSlater  Here in this problem statement it is said "On each step, after selection action $A_t$ and receiving the reward $A_t$, the action preferences are updated by...:".  I am confused because after you apply $A_t$, you obtain reward $R_{t+1}$. Hence. why in the update equation 2.12, do we not have $R_{t+1}$ instead of $R_t$?
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-25T19:18:17.807Z
Answers.Text: @DSPinfinity The choice of labelling state, action, reward in a trajectory $s_t, a_t, r_t$ or $s_t, a_t, r_{t+1}$ is one of convention, with the majority of sources I have seen using the latter. Something to be aware of when using different learning sources, as it will change the notation in some of the equations, proofs etc
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-25T19:32:52.740Z
Answers.Text: @NeilSlater I think Chapter 2 and Chapter 3 of Sutton-Barto are not consistent. Let me prove this. Chapter 2, page 26: "We denote the action selected on time step t as $A_t$ and the corresponding reward as $R_t$". Chapter 3:, page 48, Figure 3.1: "One time step later, in part as a consequence of its action, the agent receives a numerical reward of $R_{t+1}$". Do you agree on this consistency as well?
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-25T19:36:03.003Z
Answers.Text: @DSPinfinity Well chapter 2 is about bandits, and time steps are entirely separate interactions with the bandit environment. It would make no sense at all to have action $a_t$ associated with reward $r_{t+1}$ when talking about bandits. I missed that in my comment above that the source was S&B
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-25T19:38:37.480Z
Answers.Text: Perfect answer, Now clear! Thank you so much.
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-30T18:37:01.057Z
Answers.LastActivityDate: 2022-04-30T19:22:58.583Z
Answers.LastEditDate: 2022-04-30T19:22:58.583Z
Answers.UserDisplayName: Neil Slater
Body: <p>Sutton-Barto (Section  2.8-Gradient Bandit Algorithms, page 37):</p>
  <p><a href="https://i.sstatic.net/EjtGR.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/EjtGR.png" alt="enter image description here" /></a></p>
  <p>Question: is <span class="math-container">$\bar{R}_t$</span>  average of all rewards  or  average of  rewards corresponding to <span class="math-container">$A_t$</span>?</p>
  
Tags: <reinforcement-learning><sutton-barto><multi-armed-bandits>
Title: Gradient bandit algorithm: is $\bar{R}_t$ average of all rewards or average of rewards corresponding to $A_t$?
UserDisplayName: user3489173
ThreadText: # Gradient bandit algorithm: is $\bar{R}_t$ average of all rewards or average of rewards corresponding to $A_t$?
  
  (Asked by: username_0 on 2022-04-30)
  
  Sutton-Barto (Section 2.8-Gradient Bandit Algorithms, page 37):
  
  [![enter image description here](https://i.sstatic.net/EjtGR.png)](https://i.sstatic.net/EjtGR.png)
  
  Question: is $\bar{R}\_t$ average of all rewards or average of rewards corresponding to $A\_t$?
  
  **Tags:** reinforcement-learning, sutton-barto, multi-armed-bandits
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-04-30. Score: 2
  
  > Question: is $\bar{R}\_t$ average of all rewards or average of rewards corresponding to $A\_t$?
  
  It is average of all rewards seen so far. Usually a rolling recent average, so it slowly adapts to expected results from the current policy. But a mean value is good too.
  
  It is important that the value does not depend on the action choice, otherwise the gradient calculation will not be correct. In fact you can use pretty much *any* baseline provided it follows this rule, including fixed value guesses. Some baselines work out better than others in practice.
  
  If you have a contextual bandit (or an RL problem using a policy gradient method) then a good baseline choice might be average reward seen from the current state (note this depends on the *policy*, but not directly on the current action choice which might be an exploratory action). This leads to the various algorithms that use *advantage* - the difference between Q(s,a) and V(s) - as the scaling to $\nabla \text{log}(\pi(a|s))$.
  
  > **Comments:**
  > **username_0**: @ username_1 username_1, in the book it says "which can be computed incrementally as described in Section 2.4 (or Section 2.5 if the problem is nonstationary)". However, in sections 2.4 and 2.5, we calculate average of rewards for a given action, not the average of all rewards seen so far. That was the reason why I am confused.  
  > **username_1**: @username_0 I think you are expected to extract the averaging technique from that reference, not what was being averaged.  
  > **username_2**: @username_1  Here in this problem statement it is said "On each step, after selection action $A_t$ and receiving the reward $A_t$, the action preferences are updated by...:".  I am confused because after you apply $A_t$, you obtain reward $R_{t+1}$. Hence. why in the update equation 2.12, do we not have $R_{t+1}$ instead of $R_t$?  
  > **username_1**: @username_2 The choice of labelling state, action, reward in a trajectory $s_t, a_t, r_t$ or $s_t, a_t, r_{t+1}$ is one of convention, with the majority of sources I have seen using the latter. Something to be aware of when using different learning sources, as it will change the notation in some of the equations, proofs etc  
  > **username_2**: @username_1 I think Chapter 2 and Chapter 3 of Sutton-Barto are not consistent. Let me prove this. Chapter 2, page 26: "We denote the action selected on time step t as $A_t$ and the corresponding reward as $R_t$". Chapter 3:, page 48, Figure 3.1: "One time step later, in part as a consequence of its action, the agent receives a numerical reward of $R_{t+1}$". Do you agree on this consistency as well?  
  > **username_1**: @username_2 Well chapter 2 is about bandits, and time steps are entirely separate interactions with the bandit environment. It would make no sense at all to have action $a_t$ associated with reward $r_{t+1}$ when talking about bandits. I missed that in my comment above that the source was S&B  
  > **username_2**: Perfect answer, Now clear! Thank you so much.  
  
  ---
  
  