Answers.Body: <p>In some cases we may wish to have a discount factor <span class="math-container">$\gamma_t$</span> which depends on time <span class="math-container">$t$</span> (or depends on state <span class="math-container">$s_t$</span> and/or action <span class="math-container">$a_t$</span>, leading to an indirect dependence on time <span class="math-container">$t$</span>). Indeed we do not usually do this, but <a href="https://arxiv.org/abs/1902.02893" rel="noreferrer">it does happen sometimes</a>.</p>
  
  <p>I guess that, from a theoretical point of view, it was very easy of the authors to make their algorithm more flexible/general and also support this (somewhat rare) case of time-varying discount factor. If it had been very complicated for them to support this option, they may have chosen not to; but if it's trivial to do so, well, why not?</p>
  
  <p>Practical implementations will often indeed ignore that possibility if they're not using it, and can avoid including <span class="math-container">$\gamma_t$</span> values in the replay buffer altogether if it is known to be a constant <span class="math-container">$\gamma_t = \gamma$</span> for all <span class="math-container">$t$</span>. As far as I can see, in the experiments discussed in this paper they also only used a fixed, constant <span class="math-container">$\gamma$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-31T12:11:52.743Z
Answers.Text: Thanks for your answer. I do recall in Sutton & Barto the generalization to a variable gamma. But I rarely ever see it used (maybe I’ve never seen a real life implementation that wasn’t for research). I guess I forgot!
Answers.UserDisplayName: Hanzy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-31T12:28:15.943Z
Answers.Text: @Hanzy You may also sometimes see similar things in papers for variable $\lambda$ (in Sarsa($\lambda$) and other TD($\lambda$) algorithms) and/or variable learning rates $\alpha$ (although I suppose having time-varying, in particular decreasing, learning rates is quite a bit more common than variable versions of the other things)
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-31T08:26:08.763Z
Answers.LastActivityDate: 2019-05-31T08:26:08.763Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>In the <a href="https://arxiv.org/pdf/1511.05952.pdf" rel="nofollow noreferrer">original prioritized experience replay paper</a>, the authors track <span class="math-container">$\gamma_t$</span> in every state transition tuple (see line 6 in algorithm below):</p>
  
  <p><a href="https://i.sstatic.net/dSnQZ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/dSnQZ.png" alt="algorithm"></a></p>
  
  <p>Why do the authors track this at every time step?  Also, many blog posts and implementations leave this out (including I believe the <a href="https://github.com/openai/baselines/blob/master/baselines/deepq/replay_buffer.py" rel="nofollow noreferrer">OpenAI implementation</a> on github).</p>
  
  <p>Can someone explain explicitly how <span class="math-container">$\gamma_t$</span> is used in this algorithm?</p>
  
  <p><strong>Note</strong>: I understand the typical use of <span class="math-container">$\gamma$</span> as a discount factor. But typically gamma remains fixed. Which is why I’m curious as to the need to track it. </p>
  
Tags: <dqn><deep-rl><experience-replay>
Title: Why do authors track $\gamma_t$ in Prioritized Experience Replay Paper?
UserDisplayName: Hanzy
ThreadText: # Why do authors track $\gamma_t$ in Prioritized Experience Replay Paper?
  
  (Asked by: username_0 on 2019-05-31)
  
  In the [original prioritized experience replay paper](https://arxiv.org/pdf/1511.05952.pdf), the authors track $\gamma\_t$ in every state transition tuple (see line 6 in algorithm below):
  
  [![algorithm](https://i.sstatic.net/dSnQZ.png)](https://i.sstatic.net/dSnQZ.png)
  
  Why do the authors track this at every time step? Also, many blog posts and implementations leave this out (including I believe the [OpenAI implementation](https://github.com/openai/baselines/blob/master/baselines/deepq/replay_buffer.py) on github).
  
  Can someone explain explicitly how $\gamma\_t$ is used in this algorithm?
  
  **Note**: I understand the typical use of $\gamma$ as a discount factor. But typically gamma remains fixed. Which is why I’m curious as to the need to track it.
  
  **Tags:** dqn, deep-rl, experience-replay
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-05-31. Score: 4 | Accepted answer
  
  In some cases we may wish to have a discount factor $\gamma\_t$ which depends on time $t$ (or depends on state $s\_t$ and/or action $a\_t$, leading to an indirect dependence on time $t$). Indeed we do not usually do this, but [it does happen sometimes](https://arxiv.org/abs/1902.02893).
  
  I guess that, from a theoretical point of view, it was very easy of the authors to make their algorithm more flexible/general and also support this (somewhat rare) case of time-varying discount factor. If it had been very complicated for them to support this option, they may have chosen not to; but if it's trivial to do so, well, why not?
  
  Practical implementations will often indeed ignore that possibility if they're not using it, and can avoid including $\gamma\_t$ values in the replay buffer altogether if it is known to be a constant $\gamma\_t = \gamma$ for all $t$. As far as I can see, in the experiments discussed in this paper they also only used a fixed, constant $\gamma$.
  
  > **Comments:**
  > **username_0**: Thanks for your answer. I do recall in Sutton & Barto the generalization to a variable gamma. But I rarely ever see it used (maybe I’ve never seen a real life implementation that wasn’t for research). I guess I forgot!  
  > **username_1**: @username_0 You may also sometimes see similar things in papers for variable $\lambda$ (in Sarsa($\lambda$) and other TD($\lambda$) algorithms) and/or variable learning rates $\alpha$ (although I suppose having time-varying, in particular decreasing, learning rates is quite a bit more common than variable versions of the other things)  
  
  ---
  
  