Body: <p>Suppose I want to find out the Q value of a particular state <span class="math-container">$s$</span> bu doing action <span class="math-container">$a$</span> at a particular timestep <span class="math-container">$t$</span>. I know that the Q-value when the discount factor is given by,
  <span class="math-container">$$Q(a,s)=E_{\pi}\big[r_t+\gamma r_{t+1}+\gamma ^2r_{t+2}+..\big|s_t=s~~a_t=a\big]$$</span>
  If the  discount factor is not fixed, say it is  given by <span class="math-container">$\gamma(t)=e^{-t}$</span>. Then is the Q-value given by
  <span class="math-container">$$Q(a,s)=E_{\pi}\big[r_t+\gamma(t)r_{t+1}+\gamma(t)^2r_{t+2}+..\big|s_t=s~~a_t=a\big]$$</span> or is it given by
  <span class="math-container">$$Q(a,s)=E_{\pi}\big[r_t+\gamma(t+1)r_{t+1}+\gamma(t+2)^2r_{t+2}+..\big|s_t=s~~a_t=a\big]$$</span></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-11T18:28:46.067Z
Comments.Text: This seems similar to *eligibility traces* $e$ used in backward-mode TD learning. Basically, the $e$ values decay in time being time-dependent.
Comments.UserDisplayName: Luca Anzalone
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-26T07:49:06.793Z
Comments.Text: What is your motivation for having $\gamma = f(t)$, as opposed to say $\gamma = f(s)$? There are problems with definition using a function of time (independently of state - it is OK to have time as one of the properties of the state, it makes the equations work consistently), which I can go into if/when I write an answer. It would be helpful to know why you want to do this so I can suggest an alternative that may work for you
Comments.UserDisplayName: Neil Slater
Tags: <reinforcement-learning><definitions><value-functions><discount-factor>
Title: What is the definition of Q when the discount factor depends on time?
UserDisplayName: Asher2211
ThreadText: # What is the definition of Q when the discount factor depends on time?
  
  (Asked by: username_0 on 2023-05-11)
  
  Suppose I want to find out the Q value of a particular state $s$ bu doing action $a$ at a particular timestep $t$. I know that the Q-value when the discount factor is given by,
  $$Q(a,s)=E\_{\pi}\big[r\_t+\gamma r\_{t+1}+\gamma ^2r\_{t+2}+..\big|s\_t=s~~a\_t=a\big]$$
  If the discount factor is not fixed, say it is given by $\gamma(t)=e^{-t}$. Then is the Q-value given by
  $$Q(a,s)=E\_{\pi}\big[r\_t+\gamma(t)r\_{t+1}+\gamma(t)^2r\_{t+2}+..\big|s\_t=s~~a\_t=a\big]$$ or is it given by
  $$Q(a,s)=E\_{\pi}\big[r\_t+\gamma(t+1)r\_{t+1}+\gamma(t+2)^2r\_{t+2}+..\big|s\_t=s~~a\_t=a\big]$$
  
  **Tags:** reinforcement-learning, definitions, value-functions, discount-factor
  
  > **Comments:**
  > **username_1**: This seems similar to *eligibility traces* $e$ used in backward-mode TD learning. Basically, the $e$ values decay in time being time-dependent.  
  > **username_2**: What is your motivation for having $\gamma = f(t)$, as opposed to say $\gamma = f(s)$? There are problems with definition using a function of time (independently of state - it is OK to have time as one of the properties of the state, it makes the equations work consistently), which I can go into if/when I write an answer. It would be helpful to know why you want to do this so I can suggest an alternative that may work for you  
  
  ---
  
  ## 0 answers
  
  