Answers.Body: <p>Take a look at both the question (which tells you how to prove your stuff - simply replace the advantage function with your function <span class="math-container">$f$</span>) and the answer (which explains why the expectation over the trajectory is also the expectation of state-action pairs) in <a href="https://ai.stackexchange.com/a/46671/85219">https://ai.stackexchange.com/a/46671/85219</a>.</p>
  <p>Let me know if it is still not clear (and what isn't), I will update this post to include more details if needed.</p>
  <p>++++++++++++++++++++++++++++++++++++++++++++++
  +++++++++++++++++++++++++++++++++++++++++++++++</p>
  <p><strong>Edit: About expectation over trajectory</strong></p>
  <p>First, let's note that, by using <span class="math-container">$p(s_{t+1}, a_{t}|s_{t}) = \pi(a_{t}|s_{t})P(s_{t+1}|s_{t}, a_{t})$</span>, we have:</p>
  <p><span class="math-container">$\mathrm{Pr}^{\pi}_{\mu}(\tau) = \mu(s_{0})\pi(a_{0}|s_{0})P(s_{1}|s_{0}, a_{0})\cdots = \mu(s_{0})p(s_{1}, a_{0}|s_{0})\cdots = \mu(s_{0})\prod\limits^{\infty}_{t=0} p(s_{t+1}, a_{t}|s_{t})$</span></p>
  <p>Since <span class="math-container">$\tau = \{s_{0}, a_{0}, s_{1}, a_{1}, \cdots, s_{t}, a_{t}, \cdots\}$</span>, we have that:</p>
  <p><span class="math-container">$\mathrm{Pr}^{\pi}_{\mu}(\tau) = p(s_{0}, a_{0}, s_{1}, a_{1}, \cdots, s_{t}, a_{t}, \cdots)$</span></p>
  <p>which is a joint distribution. Now, at time step <span class="math-container">$t$</span>, we will have a state-action distribution <span class="math-container">$p(s_{t}, a_{t})$</span> which is clearly a marginal distribution and is given by:</p>
  <p><span class="math-container">$p(s_{t}, a_{t}) = \int p(s_{0}, a_{0}, s_{1}, a_{1}, \cdots, s_{t}, a_{t}, \cdots) ds_{0}da_{0}ds_{1}da_{1}\cdots ds_{t-1}da_{t-1}ds_{t+1}da_{t+1}\cdots$</span>
  <span class="math-container">$p(s_{t}, a_{t}) = \int \mu(s_{0})\prod\limits^{\infty}_{t=0} p(s_{t+1}, a_{t}|s_{t}) ds_{0}da_{0}ds_{1}da_{1}\cdots ds_{t-1}da_{t-1}ds_{t+1}da_{t+1}\cdots$</span></p>
  <p>Note that in the integral we skip the time step <span class="math-container">$t$</span>, i.e. there is not integral over <span class="math-container">$ds_{t}da_{t}$</span>. Now, in order to work with this, let's consider, for simplicity, a finite time step <span class="math-container">$T$</span> instead of <span class="math-container">$\infty$</span> (we can always come back by taking the limit as <span class="math-container">$T$</span> goes to infinity). So:</p>
  <p><span class="math-container">$p(s_{t}, a_{t}) = \int \mu(s_{0})\prod\limits^{T}_{t=0} p(s_{t+1}, a_{t}|s_{t}) ds_{0}da_{0}ds_{1}da_{1}\cdots ds_{t-1}da_{t-1}ds_{t+1}da_{t+1}\cdots ds_{T}da_{T}ds_{T+1}$</span></p>
  <p>This can be rewritten as:</p>
  <p><span class="math-container">$p(s_{t}, a_{t}) = \int \mu(s_{0}) \left[ \int p(s_{1},a_{0}|s_{0}) \left[ \cdots \left[ \int p(s_{t}, a_{t-1}|s_{t-1})  \left[ \int p(s_{t+1}, a_{t}|s_{t}) \left[ \cdots \left[ \int p(s_{T}, a_{T-1}|s_{T-1}) \left[ \int p(s_{T+1}, a_{T}|s_{T}) da_{T}ds_{T+1} \right] da_{T-1}ds_{T} \right] \cdots \right] ds_{t+1} \right] da_{t-1} \right] \cdots \right] da_{0}ds_{1} \right] ds_{0} $</span></p>
  <p>Starting with the time step <span class="math-container">$T$</span> and the middle integral, we can work our way backward (noting that the time step <span class="math-container">$t$</span> is not integrated). Since <span class="math-container">$\int p(s_{k+1}, a_{k}|s_{k}) da_{k}ds_{k+1} = 1$</span>, we get the sequence:</p>
  <p><span class="math-container">$p(s_{t}, a_{t}) = \int \mu(s_{0}) \left[ \int p(s_{1}, a_{0}|s_{0}) \left[ \cdots \left[ \int p(s_{t}, a_{t-1}|s_{t-1})  \left[ \int p(s_{t+1}, a_{t}|s_{t}) \left[ \cdots \left[ \int p(s_{T}, a_{T-1}|s_{T-1}) \left[ 1 \right] da_{T-1}ds_{T} \right] \cdots \right] ds_{t+1} \right] da_{t-1} \right] \cdots \right] da_{0}ds_{1} \right] ds_{0} $</span></p>
  <p><span class="math-container">$p(s_{t}, a_{t}) = \int \mu(s_{0}) \left[ \int p(s_{1}, a_{0}|s_{0}) \left[ \cdots \left[ \int p(s_{t}, a_{t-1}|s_{t-1})  \left[ \int p(s_{t+1}, a_{t}|s_{t}) \left[ \cdots \left[ 1 \right] \cdots \right] ds_{t+1} \right] da_{t-1} \right] \cdots \right] da_{0}ds_{1} \right] ds_{0} $</span></p>
  <p><span class="math-container">$p(s_{t}, a_{t}) = \int \mu(s_{0}) \left[ \int p(s_{1}, a_{0}|s_{0}) \left[ \cdots \left[ \int p(s_{t}, a_{t-1}|s_{t-1})  \left[ \int p(s_{t+1}, a_{t}|s_{t}) ds_{t+1} \right] da_{t-1} \right] \cdots \right] da_{0}ds_{1} \right] ds_{0} $</span></p>
  <p>Now, <span class="math-container">$\int p(s_{t+1}, a_{t}|s_{t}) ds_{t+1} = \int P(s_{t+1}|s_{t}, a_{t}) \pi(a_{t}|s_{t}) ds_{t+1}$</span> is not equal to <span class="math-container">$1$</span> as the other time steps integrals from <span class="math-container">$T$</span> to <span class="math-container">$t+1$</span> did. Instead, this gives the marginal distribution <span class="math-container">$\pi(a_{t}|s_{t})$</span>. Hence,</p>
  <p><span class="math-container">$p(s_{t}, a_{t}) = \int \mu(s_{0}) \left[ \int p(s_{1}, a_{0}|s_{0}) \left[ \cdots \left[ \int p(s_{t}, a_{t-1}|s_{t-1})  \left[ \pi(a_{t}|s_{t}) \right] da_{t-1} \right] \cdots \right] da_{0}ds_{1} \right] ds_{0} $</span></p>
  <p>which simplifies to:</p>
  <p><span class="math-container">$p(s_{t}, a_{t}) = \int \mu(s_{0}) p(s_{1}, a_{0}|s_{0}) \cdots p(s_{t}, a_{t-1}|s_{t-1}) \pi(a_{t}|s_{t}) da_{t-1}\cdots da_{0}ds_{1} ds_{0} $</span></p>
  <p>or more compactly:</p>
  <p><span class="math-container">$p(s_{t}, a_{t}) = \pi(a_{t}|s_{t}) \int \left[ \mu(s_{0})\prod\limits^{t-1}_{k=0} p(s_{k+1}, a_{k}|s_{k}) \right] ds_{0}da_{0}ds_{1}da_{1}\cdots ds_{t-1}da_{t-1}$</span></p>
  <p>where we are able to move <span class="math-container">$\pi(a_{t}|s_{t})$</span> outside the integral because, as said before, the time step <span class="math-container">$t$</span> is not integrated. Now, note that the result of the integral will be only a function of <span class="math-container">$s_{t}$</span>, because everything else is integrated over. We can therefore give it a name as <span class="math-container">$d^{\pi}_{\mu}(s_{t})$</span> and therefore:</p>
  <p><span class="math-container">$p(s_{t}, a_{t}) = \pi(a_{t}|s_{t}) d^{\pi}_{\mu}(s_{t})$</span></p>
  <hr />
  <hr />
  <p><strong>Note that all of the above is just the proof that</strong>:</p>
  <p><span class="math-container">$p(s_{t}, a_{t}) = \int \mu(s_{0})\prod\limits^{T}_{t=0} p(s_{t+1}, a_{t}|s_{t}) ds_{0}da_{0}ds_{1}da_{1}\cdots ds_{t-1}da_{t-1}ds_{t+1}da_{t+1}\cdots ds_{T}da_{T}ds_{T+1} = \int \left[ \mu(s_{0})\prod\limits^{t-1}_{k=0} p(s_{k+1}, a_{k}|s_{k}) ds_{0}da_{0}ds_{1}da_{1}\cdots ds_{t-1}da_{t-1} \right] \int \left[ \prod\limits^{\infty}_{k=t} p(s_{k+1}, a_{k}|s_{k}) ds_{t+1}da_{t+1}\cdots \right] = d^{\pi}_{\mu}(s_{t}) \pi(a_{t}|s_{t})$</span></p>
  <p>I could have just written this line and be done with it, but giving more details will, I hope, be more helpful. But if you can't follow the detailed derivation, just start from this.</p>
  <hr />
  <hr />
  <p>Obviously, the time step <span class="math-container">$t$</span> is completely arbitrary, meaning that the expression for the joint distribution <span class="math-container">$p(s_{t}, a_{t})$</span> is valid for every time step <span class="math-container">$t\in\{0, \cdots, T\}$</span>. So, our original joint distribution over the trajectory can be written as:</p>
  <p><span class="math-container">$\mathrm{Pr}^{\pi}_{\mu}(\tau) = p(s_{0}, a_{0}, s_{1}, a_{1}, \cdots, s_{t}, a_{t}, \cdots, s_{T}, a_{T}, s_{T+1}) = \left[\prod_{t=0}^{T} p(s_{t}, a_{t})\right] P(s_{T+1}|s_{T}, a_{T})$</span></p>
  <p>or simply:</p>
  <p><span class="math-container">$\mathrm{Pr}^{\pi}_{\mu}(\tau) = \left[\prod_{t=0}^{T} d^{\pi}_{\mu}(s_{t}) \pi(a_{t}|s_{t}) \right] P(s_{T+1}|s_{T}, a_{T})$</span></p>
  <p>From which we have:</p>
  <p><span class="math-container">$\mathbb{E}_{\tau\sim\mathrm{Pr}^{\pi}_{\mu}}\left[\sum\limits_{t=0}^{T} \gamma^{t} f(s_{t}, a_{t})\right] = \int \left[\prod_{t=0}^{T} d^{\pi}_{\mu}(s_{t}) \pi(a_{t}|s_{t}) \right] P(s_{T+1}|s_{T}, a_{T})\left[\sum\limits_{t=0}^{T} \gamma^{t} f(s_{t}, a_{t})\right]ds_{0}da_{0}ds_{1}da_{1}\cdots ds_{T}da_{T}ds_{T+1}$</span></p>
  <p><span class="math-container">$\mathbb{E}_{\tau\sim\mathrm{Pr}^{\pi}_{\mu}}\left[\sum\limits_{t=0}^{T} \gamma^{t} f(s_{t}, a_{t})\right] = \int \left[\prod_{t=0}^{T} d^{\pi}_{\mu}(s_{t}) \pi(a_{t}|s_{t}) \right] \left[\sum\limits_{t=0}^{T} \gamma^{t} f(s_{t}, a_{t})\right]ds_{0}da_{0}ds_{1}da_{1}\cdots ds_{T}da_{T}$</span></p>
  <p>This is equivalent to having <span class="math-container">$\int p(x)p(y)[\gamma_{x}f(x) + \gamma_{y}f(y)]dxdy$</span> which simply gives <span class="math-container">$\int \gamma_{x}p(x)f(x)dx + \int \gamma_{y}p(y)f(y)dy$</span>. Hence:</p>
  <p><span class="math-container">$\mathbb{E}_{\tau\sim\mathrm{Pr}^{\pi}_{\mu}}\left[\sum\limits_{t=0}^{T} \gamma^{t} f(s_{t}, a_{t})\right] = \sum\limits_{t=0}^{T} \gamma^{t} \int d^{\pi}_{\mu}(s_{t}) \pi(a_{t}|s_{t}) f(s_{t}, a_{t}) ds_{t}da_{t} = \sum\limits_{t=0}^{T} \gamma^{t} \mathbb{E}_{s_{t}\sim d^{\pi}_{\mu}(s_{t}), a_{t}\sim\pi(a_{t}|s_{t})}\left[ f(s_{t}, a_{t}) \right]$</span></p>
  <p>Now, note that inside the integral <span class="math-container">$s_{t}$</span> and <span class="math-container">$a_{t}$</span> are dummy variables. Which means they can be freely renamed to for example <span class="math-container">$s$</span> and <span class="math-container">$a$</span>:</p>
  <p><span class="math-container">$\mathbb{E}_{\tau\sim\mathrm{Pr}^{\pi}_{\mu}}\left[\sum\limits_{t=0}^{T} \gamma^{t} f(s_{t}, a_{t})\right] = \sum\limits_{t=0}^{T} \gamma^{t} \int d^{\pi}_{\mu}(s) \pi(a|s) f(s, a) ds da = \sum\limits_{t=0}^{T} \gamma^{t} \mathbb{E}_{s\sim d^{\pi}_{\mu}(s), a\sim\pi(a|s)}\left[ f(s, a) \right]$</span></p>
  <p><span class="math-container">$\mathbb{E}_{\tau\sim\mathrm{Pr}^{\pi}_{\mu}}\left[\sum\limits_{t=0}^{T} \gamma^{t} f(s_{t}, a_{t})\right] = \frac{1}{1-\gamma} \mathbb{E}_{s\sim d^{\pi}_{\mu}(s), a\sim\pi(a|s)}\left[ f(s, a) \right]$</span> CQFD</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-12T07:35:34.487Z
Answers.Text: First, thank you for helping out herre! I feel like I get the argument that is provided there intuitevely, but would it be possible to write it down in mathematical notation why it is true that the expectation over the trajectory is also the expectation of state-action pairs - I think understanding this is the next step I need to continue my proof. I am thinking that the fact has something to do with the markov property, but I am not sure.
Answers.UserDisplayName: craaaft
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-12T10:57:07.313Z
Answers.Text: @craaaft I have made the edit to add the mathematical derivation leading to the expectation over the state-action pairs.
Answers.UserDisplayName: Ahnel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-13T06:24:02.890Z
Answers.Text: Thank you so much! I will check it as an answer, but I will need some time to read it.
Answers.UserDisplayName: craaaft
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-12T02:07:55.533Z
Answers.LastActivityDate: 2024-09-12T10:56:00.680Z
Answers.LastEditDate: 2024-09-12T10:56:00.680Z
Answers.UserDisplayName: Ahnel
Body: <p>I would like to prove the following equation, which pops up everywhery in RL and I would like to have a clean proof of it. It kind of gives a representation as a trajectory vs. a representaiton in terms of state-action pairs.</p>
  <p><span class="math-container">\begin{equation}
  \mathbb{E}_{\tau \sim \mathrm{Pr}_{\mu}^\pi} \left[ \sum_{t=0}^{\infty} \gamma^t f(s_t, a_t) \right] = \frac{1}{1 - \gamma} \mathbb{E}_{s \sim d_{\mu}^\pi} \mathbb{E}_{a \sim \pi(\cdot|s)} [f(s, a)].
  \end{equation}</span></p>
  <p>where <span class="math-container">$\mathrm{Pr}_\mu^\pi(\tau) = \mu(s_0)\pi(a_0|s_0)P(s_1|s_0,a_0)\pi(a_1|s_1)\cdots
  $</span> I started the following way</p>
  <p><span class="math-container">\begin{equation}
  \mathbb{E}_{\tau \sim \mathrm{Pr}_{\mu}^\pi} \left[ \sum_{t=0}^{\infty} \gamma^t f(s_t, a_t) \right] = \sum_{\tau}\mathrm{Pr}_\mu^\pi(\tau)\sum_{t=0}^{\infty} \gamma^t f(s_t, a_t) = \sum_{\tau}\mu(s_0)\pi(a_0|s_0)P(s_1|s_0,a_0)\pi(a_1|s_1)\cdots\sum_{t=0}^{\infty} \gamma^t f(s_t, a_t).
  \end{equation}</span></p>
  <p>But here I am getting lost on how to proceed with the proof. Could anyone help me out? Also is there a way to visually explain why the expectation over the trajectory is also the expectation of state-action pairs?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-09-11T20:28:52.837Z
Comments.Text: not 100% sure but to answer "why the expectation over the trajectory is also the expectation of state-action pairs?" I would assume this is related to the steady state distribution you get in an ergodic MDP.
Comments.UserDisplayName: David
Tags: <reinforcement-learning><math><policy-gradients><proofs><notation>
Title: Proof of representation as a trajectory is the same as a representation in terms of state-action pairs
UserDisplayName: craaaft
ThreadText: # Proof of representation as a trajectory is the same as a representation in terms of state-action pairs
  
  (Asked by: username_0 on 2024-09-11)
  
  I would like to prove the following equation, which pops up everywhery in RL and I would like to have a clean proof of it. It kind of gives a representation as a trajectory vs. a representaiton in terms of state-action pairs.
  
  \begin{equation}
  \mathbb{E}\_{\tau \sim \mathrm{Pr}\_{\mu}^\pi} \left[ \sum\_{t=0}^{\infty} \gamma^t f(s\_t, a\_t) \right] = \frac{1}{1 - \gamma} \mathbb{E}\_{s \sim d\_{\mu}^\pi} \mathbb{E}\_{a \sim \pi(\cdot|s)} [f(s, a)].
  \end{equation}
  
  where $\mathrm{Pr}\_\mu^\pi(\tau) = \mu(s\_0)\pi(a\_0|s\_0)P(s\_1|s\_0,a\_0)\pi(a\_1|s\_1)\cdots
  $ I started the following way
  
  \begin{equation}
  \mathbb{E}\_{\tau \sim \mathrm{Pr}\_{\mu}^\pi} \left[ \sum\_{t=0}^{\infty} \gamma^t f(s\_t, a\_t) \right] = \sum\_{\tau}\mathrm{Pr}\_\mu^\pi(\tau)\sum\_{t=0}^{\infty} \gamma^t f(s\_t, a\_t) = \sum\_{\tau}\mu(s\_0)\pi(a\_0|s\_0)P(s\_1|s\_0,a\_0)\pi(a\_1|s\_1)\cdots\sum\_{t=0}^{\infty} \gamma^t f(s\_t, a\_t).
  \end{equation}
  
  But here I am getting lost on how to proceed with the proof. Could anyone help me out? Also is there a way to visually explain why the expectation over the trajectory is also the expectation of state-action pairs?
  
  **Tags:** reinforcement-learning, math, policy-gradients, proofs, notation
  
  > **Comments:**
  > **username_1**: not 100% sure but to answer "why the expectation over the trajectory is also the expectation of state-action pairs?" I would assume this is related to the steady state distribution you get in an ergodic MDP.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-09-12. Score: 2 | Accepted answer
  
  Take a look at both the question (which tells you how to prove your stuff - simply replace the advantage function with your function $f$) and the answer (which explains why the expectation over the trajectory is also the expectation of state-action pairs) in <https://ai.stackexchange.com/a/46671/85219>.
  
  Let me know if it is still not clear (and what isn't), I will update this post to include more details if needed.
  
  ++++++++++++++++++++++++++++++++++++++++++++++
  +++++++++++++++++++++++++++++++++++++++++++++++
  
  **Edit: About expectation over trajectory**
  
  First, let's note that, by using $p(s\_{t+1}, a\_{t}|s\_{t}) = \pi(a\_{t}|s\_{t})P(s\_{t+1}|s\_{t}, a\_{t})$, we have:
  
  $\mathrm{Pr}^{\pi}\_{\mu}(\tau) = \mu(s\_{0})\pi(a\_{0}|s\_{0})P(s\_{1}|s\_{0}, a\_{0})\cdots = \mu(s\_{0})p(s\_{1}, a\_{0}|s\_{0})\cdots = \mu(s\_{0})\prod\limits^{\infty}\_{t=0} p(s\_{t+1}, a\_{t}|s\_{t})$
  
  Since $\tau = \{s\_{0}, a\_{0}, s\_{1}, a\_{1}, \cdots, s\_{t}, a\_{t}, \cdots\}$, we have that:
  
  $\mathrm{Pr}^{\pi}\_{\mu}(\tau) = p(s\_{0}, a\_{0}, s\_{1}, a\_{1}, \cdots, s\_{t}, a\_{t}, \cdots)$
  
  which is a joint distribution. Now, at time step $t$, we will have a state-action distribution $p(s\_{t}, a\_{t})$ which is clearly a marginal distribution and is given by:
  
  $p(s\_{t}, a\_{t}) = \int p(s\_{0}, a\_{0}, s\_{1}, a\_{1}, \cdots, s\_{t}, a\_{t}, \cdots) ds\_{0}da\_{0}ds\_{1}da\_{1}\cdots ds\_{t-1}da\_{t-1}ds\_{t+1}da\_{t+1}\cdots$
  $p(s\_{t}, a\_{t}) = \int \mu(s\_{0})\prod\limits^{\infty}\_{t=0} p(s\_{t+1}, a\_{t}|s\_{t}) ds\_{0}da\_{0}ds\_{1}da\_{1}\cdots ds\_{t-1}da\_{t-1}ds\_{t+1}da\_{t+1}\cdots$
  
  Note that in the integral we skip the time step $t$, i.e. there is not integral over $ds\_{t}da\_{t}$. Now, in order to work with this, let's consider, for simplicity, a finite time step $T$ instead of $\infty$ (we can always come back by taking the limit as $T$ goes to infinity). So:
  
  $p(s\_{t}, a\_{t}) = \int \mu(s\_{0})\prod\limits^{T}\_{t=0} p(s\_{t+1}, a\_{t}|s\_{t}) ds\_{0}da\_{0}ds\_{1}da\_{1}\cdots ds\_{t-1}da\_{t-1}ds\_{t+1}da\_{t+1}\cdots ds\_{T}da\_{T}ds\_{T+1}$
  
  This can be rewritten as:
  
  $p(s\_{t}, a\_{t}) = \int \mu(s\_{0}) \left[ \int p(s\_{1},a\_{0}|s\_{0}) \left[ \cdots \left[ \int p(s\_{t}, a\_{t-1}|s\_{t-1}) \left[ \int p(s\_{t+1}, a\_{t}|s\_{t}) \left[ \cdots \left[ \int p(s\_{T}, a\_{T-1}|s\_{T-1}) \left[ \int p(s\_{T+1}, a\_{T}|s\_{T}) da\_{T}ds\_{T+1} \right] da\_{T-1}ds\_{T} \right] \cdots \right] ds\_{t+1} \right] da\_{t-1} \right] \cdots \right] da\_{0}ds\_{1} \right] ds\_{0} $
  
  Starting with the time step $T$ and the middle integral, we can work our way backward (noting that the time step $t$ is not integrated). Since $\int p(s\_{k+1}, a\_{k}|s\_{k}) da\_{k}ds\_{k+1} = 1$, we get the sequence:
  
  $p(s\_{t}, a\_{t}) = \int \mu(s\_{0}) \left[ \int p(s\_{1}, a\_{0}|s\_{0}) \left[ \cdots \left[ \int p(s\_{t}, a\_{t-1}|s\_{t-1}) \left[ \int p(s\_{t+1}, a\_{t}|s\_{t}) \left[ \cdots \left[ \int p(s\_{T}, a\_{T-1}|s\_{T-1}) \left[ 1 \right] da\_{T-1}ds\_{T} \right] \cdots \right] ds\_{t+1} \right] da\_{t-1} \right] \cdots \right] da\_{0}ds\_{1} \right] ds\_{0} $
  
  $p(s\_{t}, a\_{t}) = \int \mu(s\_{0}) \left[ \int p(s\_{1}, a\_{0}|s\_{0}) \left[ \cdots \left[ \int p(s\_{t}, a\_{t-1}|s\_{t-1}) \left[ \int p(s\_{t+1}, a\_{t}|s\_{t}) \left[ \cdots \left[ 1 \right] \cdots \right] ds\_{t+1} \right] da\_{t-1} \right] \cdots \right] da\_{0}ds\_{1} \right] ds\_{0} $
  
  $p(s\_{t}, a\_{t}) = \int \mu(s\_{0}) \left[ \int p(s\_{1}, a\_{0}|s\_{0}) \left[ \cdots \left[ \int p(s\_{t}, a\_{t-1}|s\_{t-1}) \left[ \int p(s\_{t+1}, a\_{t}|s\_{t}) ds\_{t+1} \right] da\_{t-1} \right] \cdots \right] da\_{0}ds\_{1} \right] ds\_{0} $
  
  Now, $\int p(s\_{t+1}, a\_{t}|s\_{t}) ds\_{t+1} = \int P(s\_{t+1}|s\_{t}, a\_{t}) \pi(a\_{t}|s\_{t}) ds\_{t+1}$ is not equal to $1$ as the other time steps integrals from $T$ to $t+1$ did. Instead, this gives the marginal distribution $\pi(a\_{t}|s\_{t})$. Hence,
  
  $p(s\_{t}, a\_{t}) = \int \mu(s\_{0}) \left[ \int p(s\_{1}, a\_{0}|s\_{0}) \left[ \cdots \left[ \int p(s\_{t}, a\_{t-1}|s\_{t-1}) \left[ \pi(a\_{t}|s\_{t}) \right] da\_{t-1} \right] \cdots \right] da\_{0}ds\_{1} \right] ds\_{0} $
  
  which simplifies to:
  
  $p(s\_{t}, a\_{t}) = \int \mu(s\_{0}) p(s\_{1}, a\_{0}|s\_{0}) \cdots p(s\_{t}, a\_{t-1}|s\_{t-1}) \pi(a\_{t}|s\_{t}) da\_{t-1}\cdots da\_{0}ds\_{1} ds\_{0} $
  
  or more compactly:
  
  $p(s\_{t}, a\_{t}) = \pi(a\_{t}|s\_{t}) \int \left[ \mu(s\_{0})\prod\limits^{t-1}\_{k=0} p(s\_{k+1}, a\_{k}|s\_{k}) \right] ds\_{0}da\_{0}ds\_{1}da\_{1}\cdots ds\_{t-1}da\_{t-1}$
  
  where we are able to move $\pi(a\_{t}|s\_{t})$ outside the integral because, as said before, the time step $t$ is not integrated. Now, note that the result of the integral will be only a function of $s\_{t}$, because everything else is integrated over. We can therefore give it a name as $d^{\pi}\_{\mu}(s\_{t})$ and therefore:
  
  $p(s\_{t}, a\_{t}) = \pi(a\_{t}|s\_{t}) d^{\pi}\_{\mu}(s\_{t})$
  
  ---
  
  ---
  
  **Note that all of the above is just the proof that**:
  
  $p(s\_{t}, a\_{t}) = \int \mu(s\_{0})\prod\limits^{T}\_{t=0} p(s\_{t+1}, a\_{t}|s\_{t}) ds\_{0}da\_{0}ds\_{1}da\_{1}\cdots ds\_{t-1}da\_{t-1}ds\_{t+1}da\_{t+1}\cdots ds\_{T}da\_{T}ds\_{T+1} = \int \left[ \mu(s\_{0})\prod\limits^{t-1}\_{k=0} p(s\_{k+1}, a\_{k}|s\_{k}) ds\_{0}da\_{0}ds\_{1}da\_{1}\cdots ds\_{t-1}da\_{t-1} \right] \int \left[ \prod\limits^{\infty}\_{k=t} p(s\_{k+1}, a\_{k}|s\_{k}) ds\_{t+1}da\_{t+1}\cdots \right] = d^{\pi}\_{\mu}(s\_{t}) \pi(a\_{t}|s\_{t})$
  
  I could have just written this line and be done with it, but giving more details will, I hope, be more helpful. But if you can't follow the detailed derivation, just start from this.
  
  ---
  
  ---
  
  Obviously, the time step $t$ is completely arbitrary, meaning that the expression for the joint distribution $p(s\_{t}, a\_{t})$ is valid for every time step $t\in\{0, \cdots, T\}$. So, our original joint distribution over the trajectory can be written as:
  
  $\mathrm{Pr}^{\pi}\_{\mu}(\tau) = p(s\_{0}, a\_{0}, s\_{1}, a\_{1}, \cdots, s\_{t}, a\_{t}, \cdots, s\_{T}, a\_{T}, s\_{T+1}) = \left[\prod\_{t=0}^{T} p(s\_{t}, a\_{t})\right] P(s\_{T+1}|s\_{T}, a\_{T})$
  
  or simply:
  
  $\mathrm{Pr}^{\pi}\_{\mu}(\tau) = \left[\prod\_{t=0}^{T} d^{\pi}\_{\mu}(s\_{t}) \pi(a\_{t}|s\_{t}) \right] P(s\_{T+1}|s\_{T}, a\_{T})$
  
  From which we have:
  
  $\mathbb{E}\_{\tau\sim\mathrm{Pr}^{\pi}\_{\mu}}\left[\sum\limits\_{t=0}^{T} \gamma^{t} f(s\_{t}, a\_{t})\right] = \int \left[\prod\_{t=0}^{T} d^{\pi}\_{\mu}(s\_{t}) \pi(a\_{t}|s\_{t}) \right] P(s\_{T+1}|s\_{T}, a\_{T})\left[\sum\limits\_{t=0}^{T} \gamma^{t} f(s\_{t}, a\_{t})\right]ds\_{0}da\_{0}ds\_{1}da\_{1}\cdots ds\_{T}da\_{T}ds\_{T+1}$
  
  $\mathbb{E}\_{\tau\sim\mathrm{Pr}^{\pi}\_{\mu}}\left[\sum\limits\_{t=0}^{T} \gamma^{t} f(s\_{t}, a\_{t})\right] = \int \left[\prod\_{t=0}^{T} d^{\pi}\_{\mu}(s\_{t}) \pi(a\_{t}|s\_{t}) \right] \left[\sum\limits\_{t=0}^{T} \gamma^{t} f(s\_{t}, a\_{t})\right]ds\_{0}da\_{0}ds\_{1}da\_{1}\cdots ds\_{T}da\_{T}$
  
  This is equivalent to having $\int p(x)p(y)[\gamma\_{x}f(x) + \gamma\_{y}f(y)]dxdy$ which simply gives $\int \gamma\_{x}p(x)f(x)dx + \int \gamma\_{y}p(y)f(y)dy$. Hence:
  
  $\mathbb{E}\_{\tau\sim\mathrm{Pr}^{\pi}\_{\mu}}\left[\sum\limits\_{t=0}^{T} \gamma^{t} f(s\_{t}, a\_{t})\right] = \sum\limits\_{t=0}^{T} \gamma^{t} \int d^{\pi}\_{\mu}(s\_{t}) \pi(a\_{t}|s\_{t}) f(s\_{t}, a\_{t}) ds\_{t}da\_{t} = \sum\limits\_{t=0}^{T} \gamma^{t} \mathbb{E}\_{s\_{t}\sim d^{\pi}\_{\mu}(s\_{t}), a\_{t}\sim\pi(a\_{t}|s\_{t})}\left[ f(s\_{t}, a\_{t}) \right]$
  
  Now, note that inside the integral $s\_{t}$ and $a\_{t}$ are dummy variables. Which means they can be freely renamed to for example $s$ and $a$:
  
  $\mathbb{E}\_{\tau\sim\mathrm{Pr}^{\pi}\_{\mu}}\left[\sum\limits\_{t=0}^{T} \gamma^{t} f(s\_{t}, a\_{t})\right] = \sum\limits\_{t=0}^{T} \gamma^{t} \int d^{\pi}\_{\mu}(s) \pi(a|s) f(s, a) ds da = \sum\limits\_{t=0}^{T} \gamma^{t} \mathbb{E}\_{s\sim d^{\pi}\_{\mu}(s), a\sim\pi(a|s)}\left[ f(s, a) \right]$
  
  $\mathbb{E}\_{\tau\sim\mathrm{Pr}^{\pi}\_{\mu}}\left[\sum\limits\_{t=0}^{T} \gamma^{t} f(s\_{t}, a\_{t})\right] = \frac{1}{1-\gamma} \mathbb{E}\_{s\sim d^{\pi}\_{\mu}(s), a\sim\pi(a|s)}\left[ f(s, a) \right]$ CQFD
  
  > **Comments:**
  > **username_0**: First, thank you for helping out herre! I feel like I get the argument that is provided there intuitevely, but would it be possible to write it down in mathematical notation why it is true that the expectation over the trajectory is also the expectation of state-action pairs - I think understanding this is the next step I need to continue my proof. I am thinking that the fact has something to do with the markov property, but I am not sure.  
  > **username_2**: @username_0 I have made the edit to add the mathematical derivation leading to the expectation over the state-action pairs.  
  > **username_0**: Thank you so much! I will check it as an answer, but I will need some time to read it.  
  
  ---
  
  