<p>Suppose I want to find out the Q value of a particular state <span class="math-container">$s$</span> bu doing action <span class="math-container">$a$</span> at a particular timestep <span class="math-container">$t$</span>. I know that the Q-value when the discount factor is given by,
<span class="math-container">$$Q(a,s)=E_{\pi}\big[r_t+\gamma r_{t+1}+\gamma ^2r_{t+2}+..\big|s_t=s~~a_t=a\big]$$</span>
If the  discount factor is not fixed, say it is  given by <span class="math-container">$\gamma(t)=e^{-t}$</span>. Then is the Q-value given by
<span class="math-container">$$Q(a,s)=E_{\pi}\big[r_t+\gamma(t)r_{t+1}+\gamma(t)^2r_{t+2}+..\big|s_t=s~~a_t=a\big]$$</span> or is it given by
<span class="math-container">$$Q(a,s)=E_{\pi}\big[r_t+\gamma(t+1)r_{t+1}+\gamma(t+2)^2r_{t+2}+..\big|s_t=s~~a_t=a\big]$$</span></p>

CC BY-SA 4.0
2023-05-11T18:28:46.067Z
This seems similar to *eligibility traces* $e$ used in backward-mode TD learning. Basically, the $e$ values decay in time being time-dependent.
Luca Anzalone
CC BY-SA 4.0
2023-05-26T07:49:06.793Z
What is your motivation for having $\gamma = f(t)$, as opposed to say $\gamma = f(s)$? There are problems with definition using a function of time (independently of state - it is OK to have time as one of the properties of the state, it makes the equations work consistently), which I can go into if/when I write an answer. It would be helpful to know why you want to do this so I can suggest an alternative that may work for you
Neil Slater
<reinforcement-learning><definitions><value-functions><discount-factor>
What is the definition of Q when the discount factor depends on time?
Asher2211
# What is the definition of Q when the discount factor depends on time?

(Asked by: username_0 on 2023-05-11)

Suppose I want to find out the Q value of a particular state $s$ bu doing action $a$ at a particular timestep $t$. I know that the Q-value when the discount factor is given by,
$$Q(a,s)=E\_{\pi}\big[r\_t+\gamma r\_{t+1}+\gamma ^2r\_{t+2}+..\big|s\_t=s~~a\_t=a\big]$$
If the discount factor is not fixed, say it is given by $\gamma(t)=e^{-t}$. Then is the Q-value given by
$$Q(a,s)=E\_{\pi}\big[r\_t+\gamma(t)r\_{t+1}+\gamma(t)^2r\_{t+2}+..\big|s\_t=s~~a\_t=a\big]$$ or is it given by
$$Q(a,s)=E\_{\pi}\big[r\_t+\gamma(t+1)r\_{t+1}+\gamma(t+2)^2r\_{t+2}+..\big|s\_t=s~~a\_t=a\big]$$

**Tags:** reinforcement-learning, definitions, value-functions, discount-factor

> **Comments:**
> **username_1**: This seems similar to *eligibility traces* $e$ used in backward-mode TD learning. Basically, the $e$ values decay in time being time-dependent.  
> **username_2**: What is your motivation for having $\gamma = f(t)$, as opposed to say $\gamma = f(s)$? There are problems with definition using a function of time (independently of state - it is OK to have time as one of the properties of the state, it makes the equations work consistently), which I can go into if/when I write an answer. It would be helpful to know why you want to do this so I can suggest an alternative that may work for you  

---

## 0 answers

