Answers.Body: <p>From the paper, note 3 of section 3.1.</p>
  <blockquote>
  <p>Note that, perhaps unconventionally, we treat <span class="math-container">$\gamma$</span> as a random
  variable here, because it is zero at the final step of an episode (even
  when it is constant throughout the episode).</p>
  </blockquote>
  <p>Later, it is stated that <span class="math-container">$\mathbb{V}[\gamma]$</span> is calculated based on the entire set of observed data. So the distribution of <span class="math-container">$\gamma$</span> is I imagine, categorical with two possible values; <span class="math-container">$\gamma$</span> or <span class="math-container">$0$</span>, with probability being normalised total steps vs. number of trajectories.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-21T00:54:55.410Z
Answers.Text: So say there are 3 trajectories, so 3 times $\gamma=0$, and say each trajectory has 5 steps, so $4\times3$ times $\gamma$. $12\gamma$ and $3\times0$, is that it?
Answers.UserDisplayName: Sanyou
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-09T11:17:12.673Z
Answers.Text: @Sanyou having your same doubt, I just think that $\gamma \sim Be(p)$ where $p$ is just the ratio between the steps and final-step in your MDP (so say you have a trajectory of 5, gamma is 0 only in the last step), so say you gave 10-step long episode, $E[\gamma] = 0.9 \gamma + 0.1 \cdot 0 = 0.9 \gamma$, thus the variance is $V[\gamma] = (0.9 - \gamma)^2 * 0.9 + (0.9 - 0)^2 * 0.1$
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-20T21:27:28.087Z
Answers.LastActivityDate: 2023-07-20T21:27:28.087Z
Answers.UserDisplayName: Venna Banana
Body: <p>In the paper <a href="https://arxiv.org/pdf/2105.05347.pdf" rel="nofollow noreferrer">Return-based Scaling: Yet Another Normalisation Trick for Deep RL</a>,
  <span class="math-container">$\sigma$</span>, the scaling factor to normalize the TD error, is computed as:</p>
  <p><span class="math-container">$\sigma^2 = \mathbb{V}[R] + \mathbb{V}[\gamma]\mathbb{E}[G^2]$</span></p>
  <p>What does the variance of <span class="math-container">$\gamma$</span> means? I believe <span class="math-container">$\gamma$</span> is the discount rate for the discounted return.</p>
  
Tags: <normalisation><reward-normalization>
Title: What's $\mathbb{V}[\gamma]$ in Return-based Scaling: Yet Another Normalisation Trick for Deep RL?
UserDisplayName: Sanyou
ThreadText: # What's $\mathbb{V}[\gamma]$ in Return-based Scaling: Yet Another Normalisation Trick for Deep RL?
  
  (Asked by: username_0 on 2023-07-20)
  
  In the paper [Return-based Scaling: Yet Another Normalisation Trick for Deep RL](https://arxiv.org/pdf/2105.05347.pdf),
  $\sigma$, the scaling factor to normalize the TD error, is computed as:
  
  $\sigma^2 = \mathbb{V}[R] + \mathbb{V}[\gamma]\mathbb{E}[G^2]$
  
  What does the variance of $\gamma$ means? I believe $\gamma$ is the discount rate for the discounted return.
  
  **Tags:** normalisation, reward-normalization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-07-20. Score: 2 | Accepted answer
  
  From the paper, note 3 of section 3.1.
  
  > Note that, perhaps unconventionally, we treat $\gamma$ as a random
  > variable here, because it is zero at the final step of an episode (even
  > when it is constant throughout the episode).
  
  Later, it is stated that $\mathbb{V}[\gamma]$ is calculated based on the entire set of observed data. So the distribution of $\gamma$ is I imagine, categorical with two possible values; $\gamma$ or $0$, with probability being normalised total steps vs. number of trajectories.
  
  > **Comments:**
  > **username_0**: So say there are 3 trajectories, so 3 times $\gamma=0$, and say each trajectory has 5 steps, so $4\times3$ times $\gamma$. $12\gamma$ and $3\times0$, is that it?  
  > **username_2**: @username_0 having your same doubt, I just think that $\gamma \sim Be(p)$ where $p$ is just the ratio between the steps and final-step in your MDP (so say you have a trajectory of 5, gamma is 0 only in the last step), so say you gave 10-step long episode, $E[\gamma] = 0.9 \gamma + 0.1 \cdot 0 = 0.9 \gamma$, thus the variance is $V[\gamma] = (0.9 - \gamma)^2 * 0.9 + (0.9 - 0)^2 * 0.1$  
  
  ---
  
  