Answers.Body: <p>Not an exhaustive answer, but perhaps <a href="https://towardsdatascience.com/why-there-is-a-problem-with-the-policy-gradient-theorem-in-deep-reinforcement-learning-958d845218f1" rel="nofollow noreferrer">this blog post</a> by Alessio Russo may be helpful. In particular, he states how</p>
  <blockquote>
  <p>There is an equivalence between using a discount factor and reaching a terminal state in a Markov Decision Process. [...] Therefore, we simply need to introduce, artificially, the possibility of terminating the trajectory with a certain probability 1-γ.</p>
  </blockquote>
  <p>The topic is not 100% clear to me yet either, but I feel like it makes much more sense now why this &quot;death state&quot; is used as a model to replace the discounted state distribution.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-17T14:04:45.553Z
Answers.LastActivityDate: 2023-04-17T14:04:45.553Z
Answers.UserDisplayName: Gabriele
Answers.Body: <p>If you have three possible next states from the current state, by adding the discount factor you are introducing a fourth state. It can be a terminal state or some other state that is hidden. The probability of entering that hidden state 1-γ. The probability of all other three initial possible next state is γ.</p>
  <p>Suppose you are entering the termination state by probability 1-γ. Your policy is giving a 1-γ probability of dying, so your policy has to avoid that termination and  has to make a better choice now i.e in the current state.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-14T17:53:24.093Z
Answers.Text: This is basically the same as the Sergey Levine description, which I don’t find useful (and is why I asked the question).
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-14T04:50:20.610Z
Answers.LastActivityDate: 2023-11-14T15:48:23.863Z
Answers.LastEditDate: 2023-11-14T15:48:23.863Z
Answers.UserDisplayName: GEORGE THOMAS
Answers.Body: <p>The R.S. Sutton and A. G. Barto book says &quot;Often <span class="math-container">$\mu(s)$</span> is chosen to be the fraction of time spent in <span class="math-container">$s$</span>? But then why a discount factor is needed in the continual case?</p>
  <p>Discounting is practical when working with discounted rewards but not necessary when working with average rewards.</p>
  <p><strong>Why do may need to discount our state distribution with discounted rewards?</strong></p>
  <p>Let <span class="math-container">$\gamma \in [0,1[$</span> be the discount factor.</p>
  <p>Let the discounted state measure be defined as
  <span class="math-container">$$
  \mu(s) = \sum_{t=0}^{\infty} \gamma^t \mathbb{P}^\pi(S_t = s)
  $$</span>
  where <span class="math-container">$\mathbb{P}^\pi(S_t = s)$</span> is the probability of being in state <span class="math-container">$s$</span> at time <span class="math-container">$t$</span>.</p>
  <p><span class="math-container">$$
  \int_\mathcal{S} \mu(s) ds = \sum_{s \in \mathcal{S}} \mu(s) = \sum_{s \in \mathcal{S}} \sum_{t=0}^{\infty} \gamma^t \mathbb{P}^\pi(S_t = s) = \sum_{t=0}^{\infty} \gamma^t \sum_{s \in \mathcal{S}} \mathbb{P}^\pi(S_t = s) = \sum_{t=0}^{\infty} \gamma^t  = \frac{1}{1-\gamma}
  $$</span></p>
  <p>where <span class="math-container">$\mathcal{S}$</span> is the state space.
  Thus, to be a distribution <span class="math-container">$\mu$</span> has to be multiplied by <span class="math-container">$1 -\gamma$</span> as in [1].</p>
  <p>Note that if <span class="math-container">$\gamma = 1$</span>, then the total mass of <span class="math-container">$\mu$</span> goes to <span class="math-container">$+\infty$</span>.
  The distribution <span class="math-container">$\mu$</span> is said <em>improper</em> in this case.</p>
  <p>Also remark that,
  <span class="math-container">$$
  \mathbb{P}^\pi(S_t = s, A_t = a) = \mathbb{P}^\pi(S_t = s) \mathbb{P}^\pi(A_t = a | S_t = s) = \mathbb{P}^\pi(S_t = s) \pi(a|s)
  $$</span>
  where <span class="math-container">$\pi(a|s)$</span> is the policy.</p>
  <p>Now, remark with <span class="math-container">$r : \mathcal{S} \times \mathcal{A} \to \mathbb{R}$</span> the reward function.</p>
  <p><span class="math-container">$$
  \mathbb{E}^\pi[\sum_{t=0}^{\infty} \gamma^t r(S_t, A_t)] = \sum_{t=0}^{\infty} \gamma^t \mathbb{E}^\pi[r(S_t, A_t)] = \sum_{t=0}^{\infty} \gamma^t \sum_{s \in \mathcal{S}} \sum_{a \in \mathcal{A}} r(s,a) \mathbb{P}^\pi(S_t = s, A_t = a)
  = \sum_{s \in \mathcal{S}} \sum_{a \in \mathcal{A}} \sum_{t=0}^{\infty} \gamma^t \mathbb{P}^\pi(S_t = s) \pi(a|s)  r(s,a)
  = \sum_{s \in \mathcal{S}} \sum_{a \in \mathcal{A}} \mu(s) \pi(a|s)  r(s,a)
  = \mathbb{E}_\mu\mathbb{E}_\pi[r(S,A)]
  $$</span></p>
  <p>Let's break down what happened here.
  We started by averaging the reward of the trajectory <span class="math-container">$(S_0, A_0, S_1, A_1, \ldots)$</span> that is characterised by the distribution <span class="math-container">$\mathbb{P}^\pi$</span>. This distribution characterised a <strong>trajectory</strong> (that's why I used the subscript <span class="math-container">$t$</span> inside the expectation).</p>
  <p>On the other hand, the distribution <span class="math-container">$\mu$</span> is defined on the state-space <span class="math-container">$\mathcal{S}$</span> and characterises the probability a state is visited (that's why I do not used the subscript <span class="math-container">$t$</span> inside the expectation, to mark the time-independence of the random variable).</p>
  <p>Consequently <span class="math-container">$\mathbb{P}^\pi$</span> and <span class="math-container">$\mu$</span> are two different distributions that are defined on different spaces! The distribution <span class="math-container">$\mu$</span> averages over the time inherently.
  This kind of change of measure often appears in RL literature and are not always obvious to understand.</p>
  <p><em>Introducing the discounted measure allows in practice to sample state-action pairs <span class="math-container">$(X, A) \in \mathcal{X} \times \mathcal{A}$</span> to evaluate the objective function</em>, instead of doing trajectory-based Monte-Carlo methods. In practice, it is very convenient and allows to employ &quot;replay-buffer&quot; for instance.</p>
  <p>Going back to your question we have observed the importance of discounting in the &quot;continual case&quot; (infinite trajectories).</p>
  <p><strong>Why multiplying by <span class="math-container">$\gamma$</span> introduces a termination condition?</strong></p>
  <p>Take a probability distribution <span class="math-container">$\mu(\cdot)$</span> or <span class="math-container">$\mathcal{P}(\cdot, x, a)$</span> for some <span class="math-container">$x \in \mathcal{S}$</span> and <span class="math-container">$a \in \mathcal{A}$</span>.
  Then
  <span class="math-container">$$
  \int_\mathcal{S} \gamma \mu(s) ds = \gamma \int_\mathcal{S} \mu(s) ds = \gamma
  = \gamma \int_\mathcal{S} \mathcal{P}(s, x, a) ds
  $$</span></p>
  <p>So both distribution <span class="math-container">$\gamma \mu$</span> and <span class="math-container">$\gamma \mathcal{P}$</span> are not probability distributions anymore! They are not normalised to 1. One possible way to get a probability distribution is to normalise it OR to introduce a state <span class="math-container">$s_{\text{end}}$</span> that is absorbing and has a reward of 0 with a probability of <span class="math-container">$1-\gamma$</span>. This new MDP constructed by augmenting the state space with <span class="math-container">$s_{\text{end}}$</span> is then considered.</p>
  <hr />
  <p>For mathematicians, to work rigorously with <span class="math-container">$\mu$</span> it is always better to use its truncated version and pass to the limit since limits of measures are still measures when the limit exists.</p>
  <p>[1]: A. Agarwal et al. - Reinforcement Learning: Theory and Algorithms</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-19T02:48:03.297Z
Answers.Text: About the first step after "Now, remark with ...". Having the superscript \pi in the first expectation is weird: should it be over {s0, a0, s1, a1, ...}? This is the standard notation (see TRPO paper, definition of eta(pi)). The, what is the second expectation over? And can we pull it inside the sum like that? If it is still over the trajectory, samples have timestep, so technically we can't. Unless we are assuming stationarity of the state distribution.
Answers.UserDisplayName: Simon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-19T13:07:56.037Z
Answers.Text: Hello @Simon, Indeed there is an inconsistancy in notations with a super and subscript $\pi$. The first expectation and the second after the first inequality are strictly the same. I just corrected this typo.
  
  An expectation is an integral over a measure in a measurable space $(\Omega, \mathcal{F}, \mathbb{P})$, here $\mathbb{E}^\pi$ is the integral over $\mathbb{P}^\pi$.
  
  The series of expectations being equal to the expectation of the series is true at least when the reward are bounded, by the dominated convergence theorem https://en.wikipedia.org/wiki/Dominated_convergence_theorem.
Answers.UserDisplayName: Rémy Hosseinkhan Boucher
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-19T13:21:26.787Z
Answers.Text: The crucial part is where the "triple sum" equals the "double sum" under $\mu$. Here, by definition of your measure $\mu$, you have the equality...
  
  Your point on the stationarity is very relevant. I agree that stationarity allows dropping the time subscripts to get spatial average. However, here it is not applicable, but this approach kind of find a trick to obtain a results similar to stationarity, and it is rigorously accurate.
  
  Try the same derivation with an avg rewards criterion, under stationarity hypothesis, you will get smth similar. Many prefer the avg criterion for that reason!
Answers.UserDisplayName: Rémy Hosseinkhan Boucher
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-19T16:20:54.673Z
Answers.Text: Thanks for your reply. I am pretty sure we must assume mu stationarity. Otherwise mu(s) depends on t and it is actually written as mu^(t)(s) (see https://arxiv.org/pdf/2011.06631). What do you mean by "here it is not applicable"?
Answers.UserDisplayName: Simon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-19T16:21:05.070Z
Answers.Text: And I am also confused by your result because you start with the discounted reward criterion and end up with the average reward criterion. There should be a factor 1/(1-gamma) (page 254 or Sutton's book). Or is it being pulled in mu(s)?
Answers.UserDisplayName: Simon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-19T16:52:42.573Z
Answers.Text: I indeed start with the discounted reward criterion, and I find a formulation of this criterion under two new measures (double integral $\mathbb{E}_\mu \mathbb{E}_\pi$) for the discounted reward criterion. No stationarity is involved here, but a close approach that makes $\mu$ containing/averaging the time.
  
  In the paper you cite $\rho^{(t)} = \mathbb{P}(S_t = \cdot)$ while $\rho$ is the stationary measure (the infinite horizon average- p. 249 of Sutton and Barto but not normalised by the time). See G. Neu et al. - A Unified View of Entropy-Regularized MDP, p.4 for the stationary measure.
Answers.UserDisplayName: Rémy Hosseinkhan Boucher
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-19T16:58:05.017Z
Answers.Text: The $\mu$ defined in my answer is time-independant but no stationarity is mentioned at all right? I am not an expert, but a stationary distribution won't help to obtain an expression of the **discounted** reward criterion while the average criterion can be expressed with this stationary measure. That's why we have to craft this time-independant $\mu$ for the discounted case!:)
  
  Regarding Sutton's book, it is not very clear how those $\mu$ are defined, but the sure thing is that they are independent of time (as the stationary dist. or the discounted visitation measure I defined - check [1])!
Answers.UserDisplayName: Rémy Hosseinkhan Boucher
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-19T21:15:54.960Z
Answers.Text: Yep, I see how you connected the two formulations, and I know they are correct. But if $\mu$ is time-independent then it's stationary. Also Sutton at page 249 says that the equations hold if $\mu$ is the steady-state (=stationary) distribution. The paper you cited also assumes so in Assumption 1. That's why I said that in your answer (great answer btw) you must also assume so.
Answers.UserDisplayName: Simon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-19T22:02:47.243Z
Answers.Text: @Simon, Sutton's p.49 and Neu's paper **both use the average criterion** !! By stationary, I think we agree that we are both talking about the *invariant* measure as defined in the "Stationary Distribution" part of the Markov Chain Wikipedia [page](https://en.wikipedia.org/wiki/Markov_chain) of the underlying Markov Chain.
  
  Consequently, your statement "if $\mu$ is time-independant, then it is stationary" is rigorously wrong, I mean at least in the sense of a true invariant measure! Indeed, $\mu$ looks like a kind of "stationary" measure but it not in the context of discounted rewards!
Answers.UserDisplayName: Rémy Hosseinkhan Boucher
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-19T22:11:39.540Z
Answers.Text: @Simon, I think the point where we disagree is that you think $\mu$ is the invariant, stationary measure for the Markov chain, while I pretend that it is not.
  
  Consider $\mu$ being "some" measure that is independent of the time but which is not the stationary invariant measure, and that satisfies the equality for discounted rewards:)
  Thus, we have **constructed** a measure $\mu$, which is time independent such that our result hold.
  
  Moreover, in the literature $\mu$ for discounted reward is called a "visitation measure"
Answers.UserDisplayName: Rémy Hosseinkhan Boucher
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-19T22:16:31.383Z
Answers.Text: The notation μ is chosen to underline that μ is similar to a stationary measure.
  However, this derivation does not need the stationarity assumption since the reasoning stems from the **construction** of this measure... Ask yourself, which assumptions should hold to construct the discounted visited state measure $\mu$? However, If you choose average rewards you will observe that to get a similar result, $\mu$ must be constructed such that $\mu$ coincide with the stationary measure. (While you can also use the ergodic theorem to get the result I guess).
Answers.UserDisplayName: Rémy Hosseinkhan Boucher
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-20T05:40:20.383Z
Answers.Text: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/156299/discussion-between-simon-and-remy-hosseinkhan-boucher).
Answers.UserDisplayName: Simon
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-04T11:44:16.173Z
Answers.LastActivityDate: 2024-11-19T13:16:19.520Z
Answers.LastEditDate: 2024-11-19T13:16:19.520Z
Answers.UserDisplayName: Rémy Hosseinkhan Boucher
Body: <p>In Reinforcement Learning, it is common to use a discount factor <span class="math-container">$\gamma$</span> to give less importance to future rewards when calculating the returns.</p>
  <p>I have also seen mention of discounted state distributions. It is mentioned on page 199 of the Sutton and Barto textbook that if there is discounting then (for the state distribution) it should be treated as a form of termination, and it is implied that this can be achieved by adding a factor of <span class="math-container">$\gamma$</span> to the state transition dynamics of the MDP, so that now we have</p>
  <p><span class="math-container">$$\mu(s) = \frac{\eta(s)}{\sum_{s'} \eta(s')}\;;$$</span>
  where <span class="math-container">$\eta(s) = h(s) + \sum_{\bar{s}} \eta(\bar{s})\sum_a \pi(a|\bar{s}) \gamma p(s|\bar{s}, a)$</span> and <span class="math-container">$h(s)$</span> is the probability of the episode beginning in state <span class="math-container">$s$</span>.</p>
  <p>In my opinion, the book kind of skips over this and it is not immediately clear to me <em>why</em> we need to discount our state distribution if we have discounting in the episode.</p>
  <p>My intuition would suggest that it is because we usually take an expectation of the returns over the state distribution (and action/transition dynamics), but, if we are discounting the (future) rewards, then we should also discount the future states to give them less importance. In <a href="https://rll.berkeley.edu/deeprlcourse/f17docs/lecture_5_actor_critic_pdf.pdf" rel="nofollow noreferrer">Sergey Levine's lectures</a> he provides a brief aside that I think agrees with my intuition but in a rather unsatisfactory way -- he introduces the idea of a 'death state' that we transition into at each step with probability <span class="math-container">$1-\gamma$</span> but he does not really provide a rigorous enough justification for thinking of it this way (unless it is just a useful mental model and not supposed to be rigorous).</p>
  <p>I am wondering whether someone can provide a more detailed explanation as to why we discount the state distribution.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-11-04T11:47:57.270Z
Comments.Text: I added an answer @David
Comments.UserDisplayName: Rémy Hosseinkhan Boucher
Tags: <reinforcement-learning><markov-decision-process><discount-factor>
Title: Why do we discount the state distribution?
UserDisplayName: David
ThreadText: # Why do we discount the state distribution?
  
  (Asked by: username_0 on 2021-04-06)
  
  In Reinforcement Learning, it is common to use a discount factor $\gamma$ to give less importance to future rewards when calculating the returns.
  
  I have also seen mention of discounted state distributions. It is mentioned on page 199 of the Sutton and Barto textbook that if there is discounting then (for the state distribution) it should be treated as a form of termination, and it is implied that this can be achieved by adding a factor of $\gamma$ to the state transition dynamics of the MDP, so that now we have
  
  $$\mu(s) = \frac{\eta(s)}{\sum\_{s'} \eta(s')}\;;$$
  where $\eta(s) = h(s) + \sum\_{\bar{s}} \eta(\bar{s})\sum\_a \pi(a|\bar{s}) \gamma p(s|\bar{s}, a)$ and $h(s)$ is the probability of the episode beginning in state $s$.
  
  In my opinion, the book kind of skips over this and it is not immediately clear to me *why* we need to discount our state distribution if we have discounting in the episode.
  
  My intuition would suggest that it is because we usually take an expectation of the returns over the state distribution (and action/transition dynamics), but, if we are discounting the (future) rewards, then we should also discount the future states to give them less importance. In [Sergey Levine's lectures](https://rll.berkeley.edu/deeprlcourse/f17docs/lecture_5_actor_critic_pdf.pdf) he provides a brief aside that I think agrees with my intuition but in a rather unsatisfactory way -- he introduces the idea of a 'death state' that we transition into at each step with probability $1-\gamma$ but he does not really provide a rigorous enough justification for thinking of it this way (unless it is just a useful mental model and not supposed to be rigorous).
  
  I am wondering whether someone can provide a more detailed explanation as to why we discount the state distribution.
  
  **Tags:** reinforcement-learning, markov-decision-process, discount-factor
  
  > **Comments:**
  > **username_1**: I added an answer @username_0  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2023-04-17. Score: 1 | Accepted answer
  
  Not an exhaustive answer, but perhaps [this blog post](https://towardsdatascience.com/why-there-is-a-problem-with-the-policy-gradient-theorem-in-deep-reinforcement-learning-958d845218f1) by Alessio Russo may be helpful. In particular, he states how
  
  > There is an equivalence between using a discount factor and reaching a terminal state in a Markov Decision Process. [...] Therefore, we simply need to introduce, artificially, the possibility of terminating the trajectory with a certain probability 1-γ.
  
  The topic is not 100% clear to me yet either, but I feel like it makes much more sense now why this "death state" is used as a model to replace the discounted state distribution.
  
  
  ---
  
  ### Answer by username_3 on 2023-11-14. Score: 1
  
  If you have three possible next states from the current state, by adding the discount factor you are introducing a fourth state. It can be a terminal state or some other state that is hidden. The probability of entering that hidden state 1-γ. The probability of all other three initial possible next state is γ.
  
  Suppose you are entering the termination state by probability 1-γ. Your policy is giving a 1-γ probability of dying, so your policy has to avoid that termination and has to make a better choice now i.e in the current state.
  
  > **Comments:**
  > **username_0**: This is basically the same as the Sergey Levine description, which I don’t find useful (and is why I asked the question).  
  
  ---
  
  ### Answer by username_1 on 2024-11-04. Score: 1
  
  The R.S. Sutton and A. G. Barto book says "Often $\mu(s)$ is chosen to be the fraction of time spent in $s$? But then why a discount factor is needed in the continual case?
  
  Discounting is practical when working with discounted rewards but not necessary when working with average rewards.
  
  **Why do may need to discount our state distribution with discounted rewards?**
  
  Let $\gamma \in [0,1[$ be the discount factor.
  
  Let the discounted state measure be defined as
  $$
  \mu(s) = \sum\_{t=0}^{\infty} \gamma^t \mathbb{P}^\pi(S\_t = s)
  $$
  where $\mathbb{P}^\pi(S\_t = s)$ is the probability of being in state $s$ at time $t$.
  
  $$
  \int\_\mathcal{S} \mu(s) ds = \sum\_{s \in \mathcal{S}} \mu(s) = \sum\_{s \in \mathcal{S}} \sum\_{t=0}^{\infty} \gamma^t \mathbb{P}^\pi(S\_t = s) = \sum\_{t=0}^{\infty} \gamma^t \sum\_{s \in \mathcal{S}} \mathbb{P}^\pi(S\_t = s) = \sum\_{t=0}^{\infty} \gamma^t = \frac{1}{1-\gamma}
  $$
  
  where $\mathcal{S}$ is the state space.
  Thus, to be a distribution $\mu$ has to be multiplied by $1 -\gamma$ as in [1].
  
  Note that if $\gamma = 1$, then the total mass of $\mu$ goes to $+\infty$.
  The distribution $\mu$ is said *improper* in this case.
  
  Also remark that,
  $$
  \mathbb{P}^\pi(S\_t = s, A\_t = a) = \mathbb{P}^\pi(S\_t = s) \mathbb{P}^\pi(A\_t = a | S\_t = s) = \mathbb{P}^\pi(S\_t = s) \pi(a|s)
  $$
  where $\pi(a|s)$ is the policy.
  
  Now, remark with $r : \mathcal{S} \times \mathcal{A} \to \mathbb{R}$ the reward function.
  
  $$
  \mathbb{E}^\pi[\sum\_{t=0}^{\infty} \gamma^t r(S\_t, A\_t)] = \sum\_{t=0}^{\infty} \gamma^t \mathbb{E}^\pi[r(S\_t, A\_t)] = \sum\_{t=0}^{\infty} \gamma^t \sum\_{s \in \mathcal{S}} \sum\_{a \in \mathcal{A}} r(s,a) \mathbb{P}^\pi(S\_t = s, A\_t = a)
  = \sum\_{s \in \mathcal{S}} \sum\_{a \in \mathcal{A}} \sum\_{t=0}^{\infty} \gamma^t \mathbb{P}^\pi(S\_t = s) \pi(a|s) r(s,a)
  = \sum\_{s \in \mathcal{S}} \sum\_{a \in \mathcal{A}} \mu(s) \pi(a|s) r(s,a)
  = \mathbb{E}\_\mu\mathbb{E}\_\pi[r(S,A)]
  $$
  
  Let's break down what happened here.
  We started by averaging the reward of the trajectory $(S\_0, A\_0, S\_1, A\_1, \ldots)$ that is characterised by the distribution $\mathbb{P}^\pi$. This distribution characterised a **trajectory** (that's why I used the subscript $t$ inside the expectation).
  
  On the other hand, the distribution $\mu$ is defined on the state-space $\mathcal{S}$ and characterises the probability a state is visited (that's why I do not used the subscript $t$ inside the expectation, to mark the time-independence of the random variable).
  
  Consequently $\mathbb{P}^\pi$ and $\mu$ are two different distributions that are defined on different spaces! The distribution $\mu$ averages over the time inherently.
  This kind of change of measure often appears in RL literature and are not always obvious to understand.
  
  *Introducing the discounted measure allows in practice to sample state-action pairs $(X, A) \in \mathcal{X} \times \mathcal{A}$ to evaluate the objective function*, instead of doing trajectory-based Monte-Carlo methods. In practice, it is very convenient and allows to employ "replay-buffer" for instance.
  
  Going back to your question we have observed the importance of discounting in the "continual case" (infinite trajectories).
  
  **Why multiplying by $\gamma$ introduces a termination condition?**
  
  Take a probability distribution $\mu(\cdot)$ or $\mathcal{P}(\cdot, x, a)$ for some $x \in \mathcal{S}$ and $a \in \mathcal{A}$.
  Then
  $$
  \int\_\mathcal{S} \gamma \mu(s) ds = \gamma \int\_\mathcal{S} \mu(s) ds = \gamma
  = \gamma \int\_\mathcal{S} \mathcal{P}(s, x, a) ds
  $$
  
  So both distribution $\gamma \mu$ and $\gamma \mathcal{P}$ are not probability distributions anymore! They are not normalised to 1. One possible way to get a probability distribution is to normalise it OR to introduce a state $s\_{\text{end}}$ that is absorbing and has a reward of 0 with a probability of $1-\gamma$. This new MDP constructed by augmenting the state space with $s\_{\text{end}}$ is then considered.
  
  ---
  
  For mathematicians, to work rigorously with $\mu$ it is always better to use its truncated version and pass to the limit since limits of measures are still measures when the limit exists.
  
  [1]: A. Agarwal et al. - Reinforcement Learning: Theory and Algorithms
  
  > **Comments:**
  > **username_4**: About the first step after "Now, remark with ...". Having the superscript \pi in the first expectation is weird: should it be over {s0, a0, s1, a1, ...}? This is the standard notation (see TRPO paper, definition of eta(pi)). The, what is the second expectation over? And can we pull it inside the sum like that? If it is still over the trajectory, samples have timestep, so technically we can't. Unless we are assuming stationarity of the state distribution.  
  > **username_1**: Hello @username_4, Indeed there is an inconsistancy in notations with a super and subscript $\pi$. The first expectation and the second after the first inequality are strictly the same. I just corrected this typo.  An expectation is an integral over a measure in a measurable space $(\Omega, \mathcal{F}, \mathbb{P})$, here $\mathbb{E}^\pi$ is the integral over $\mathbb{P}^\pi$.  The series of expectations being equal to the expectation of the series is true at least when the reward are bounded, by the dominated convergence theorem https://en.wikipedia.org/wiki/Dominated_convergence_theorem.  
  > **username_1**: The crucial part is where the "triple sum" equals the "double sum" under $\mu$. Here, by definition of your measure $\mu$, you have the equality...  Your point on the stationarity is very relevant. I agree that stationarity allows dropping the time subscripts to get spatial average. However, here it is not applicable, but this approach kind of find a trick to obtain a results similar to stationarity, and it is rigorously accurate.  Try the same derivation with an avg rewards criterion, under stationarity hypothesis, you will get smth similar. Many prefer the avg criterion for that reason!  
  > **username_4**: Thanks for your reply. I am pretty sure we must assume mu stationarity. Otherwise mu(s) depends on t and it is actually written as mu^(t)(s) (see https://arxiv.org/pdf/2011.06631). What do you mean by "here it is not applicable"?  
  > **username_4**: And I am also confused by your result because you start with the discounted reward criterion and end up with the average reward criterion. There should be a factor 1/(1-gamma) (page 254 or Sutton's book). Or is it being pulled in mu(s)?  
  > **username_1**: I indeed start with the discounted reward criterion, and I find a formulation of this criterion under two new measures (double integral $\mathbb{E}_\mu \mathbb{E}_\pi$) for the discounted reward criterion. No stationarity is involved here, but a close approach that makes $\mu$ containing/averaging the time.  In the paper you cite $\rho^{(t)} = \mathbb{P}(S_t = \cdot)$ while $\rho$ is the stationary measure (the infinite horizon average- p. 249 of Sutton and Barto but not normalised by the time). See G. Neu et al. - A Unified View of Entropy-Regularized MDP, p.4 for the stationary measure.  
  > **username_1**: The $\mu$ defined in my answer is time-independant but no stationarity is mentioned at all right? I am not an expert, but a stationary distribution won't help to obtain an expression of the **discounted** reward criterion while the average criterion can be expressed with this stationary measure. That's why we have to craft this time-independant $\mu$ for the discounted case!:)  Regarding Sutton's book, it is not very clear how those $\mu$ are defined, but the sure thing is that they are independent of time (as the stationary dist. or the discounted visitation measure I defined - check [1])!  
  > **username_4**: Yep, I see how you connected the two formulations, and I know they are correct. But if $\mu$ is time-independent then it's stationary. Also Sutton at page 249 says that the equations hold if $\mu$ is the steady-state (=stationary) distribution. The paper you cited also assumes so in Assumption 1. That's why I said that in your answer (great answer btw) you must also assume so.  
  > **username_1**: @username_4, Sutton's p.49 and Neu's paper **both use the average criterion** !! By stationary, I think we agree that we are both talking about the *invariant* measure as defined in the "Stationary Distribution" part of the Markov Chain Wikipedia [page](https://en.wikipedia.org/wiki/Markov_chain) of the underlying Markov Chain.  Consequently, your statement "if $\mu$ is time-independant, then it is stationary" is rigorously wrong, I mean at least in the sense of a true invariant measure! Indeed, $\mu$ looks like a kind of "stationary" measure but it not in the context of discounted rewards!  
  > **username_1**: @username_4, I think the point where we disagree is that you think $\mu$ is the invariant, stationary measure for the Markov chain, while I pretend that it is not.  Consider $\mu$ being "some" measure that is independent of the time but which is not the stationary invariant measure, and that satisfies the equality for discounted rewards:) Thus, we have **constructed** a measure $\mu$, which is time independent such that our result hold.  Moreover, in the literature $\mu$ for discounted reward is called a "visitation measure"  
  > **username_1**: The notation μ is chosen to underline that μ is similar to a stationary measure. However, this derivation does not need the stationarity assumption since the reasoning stems from the **construction** of this measure... Ask yourself, which assumptions should hold to construct the discounted visited state measure $\mu$? However, If you choose average rewards you will observe that to get a similar result, $\mu$ must be constructed such that $\mu$ coincide with the stationary measure. (While you can also use the ergodic theorem to get the result I guess).  
  > **username_4**: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/156299/discussion-between-username_4-and-remy-hosseinkhan-username_1).  
  
  ---
  
  