Answers.Body: <p>Yes, <span class="math-container">$S_t \sim d^\pi$</span> is a nice way of saying that the states are distributed according to the state distribution induced by following <span class="math-container">$\pi$</span>. Whilst <span class="math-container">$\pi$</span> does not directly choose the next state, choosing the action (which is sampled directly from <span class="math-container">$\pi$</span>) is clearly influencing what the next state will be, so the distribution of visited states will depend upon <span class="math-container">$\pi$</span>.</p>
  <p>Generally you can just add a <span class="math-container">$t$</span> subscript but that would be on you to make sure it makes sense. Generally it is omitted to avoid overloading notation.</p>
  <p>For the reward, I would first point out that a random variable is a function. But you can write this however is easier for you -- generally there are different ways of writing this, e.g. <span class="math-container">$r_t, r(s_t, a_t), r(s_t, a_t, s_{t+1})$</span>. The first is probably the most commonly used as it is the cleanest to read but they are all equivalent (e.g. the second term is typically used in place of the final term when the environment dynamics are deterministic and thus <span class="math-container">$s_{t+1}$</span> doesn't affect the reward).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-30T08:43:47.853Z
Answers.Text: Thank you! Ok so notation in regards of the policy gradient would be the same in both as I understand. True, the RV is a function but then the reward is a function of two random variables S and A. Anyhow, thank you! I am probably overthinking all of this. I was just wondering since I couldn't find any rssource that was introducing the subject using a stochastic reward.
Answers.UserDisplayName: craaaft
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-30T09:05:39.757Z
Answers.Text: And do you have maybe any comments about my last question about conditional vs unconditional expectation? Conditional expectation is always when we want to specify some starting point as I assume, but often underneath the expectation sign there is indicaiton of sampling from the policy? But generally I think there should be?
Answers.UserDisplayName: craaaft
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-30T13:11:01.630Z
Answers.Text: you used conditional expectation when you want to assume the outcome of an RV is known. For instance, the value function is the value of the expected returns starting from state $s$, i.e. _conditioning_ that $S_t = s$. I would expect that most authors include the conditioning in the expectation tbh, I've not really come across many examples where it is omitted when it is indeed being conditioned on.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-30T17:33:41.927Z
Answers.Text: But given $S_t = s$ we have to further sample trajectories to determine the value function. Wouldn't it then make sense to write the sampling from a policy under the expectation sign additionally to conditioning it on $S_t = s$?
Answers.UserDisplayName: craaaft
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-30T17:35:02.777Z
Answers.Text: So the two ways to state the policy gradient above are equivalent right?
Answers.UserDisplayName: craaaft
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-02T07:32:15.213Z
Answers.Text: you can write this yes. e.g. you might write $v_\pi(s) = \mathbb{E}_{\tau \sim (\pi, p)} \left[ ... \right]$ where $\tau$ is the trajectory and $p$ is the environment dynamics (so we are saying the trajectory is generated according to the policy and dynamics), and the $...$ would be your usual returns and conditioning.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-02T07:35:45.153Z
Answers.Text: the two PG's look equivalent. the second is just more explicitly saying which distribution the states are sampled from, which is the state distribution induced by the policy. generally, though, this is implied rather than explicitly stated.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-30T07:11:09.400Z
Answers.LastActivityDate: 2024-08-30T07:11:09.400Z
Answers.UserDisplayName: David
Body: <p>I have some questions on notation in reinforcement learning, since it seems that different papers, books have some different notation. I am writing a thesis and wanted to have the notation consistent. I base my notationon this <a href="https://github.com/MathFoundationRL/Book-Mathematical-Foundation-of-Reinforcement-Learning" rel="nofollow noreferrer">book</a>, but then a paper like TRPO has some different kind of notation. My first question is the following - some people write <span class="math-container">$\pi$</span> under the expectation which means that they sample the trajectory from the policy</p>
  <p><span class="math-container">$$
  \nabla J(\theta) =\mathbb{E}_{\pi}[\sum_{a \in A} \pi(a|S_t)q_{\pi}(S_t,a)\ln\nabla\pi_{\theta}(a|S_t)] = \mathbb{E}_{\pi}[q_{\pi}(S_t,A_t)\ln\nabla\pi_{\theta}(A_t|S_t)] 
  $$</span></p>
  <p>is this the same as the following though, first sampling from the stationary distribution we obtain after running the policy for some long time:</p>
  <p><span class="math-container">$$
  \nabla J(\theta) = \mathbb{E}_{S_t\sim d^\pi, A_t\sim\pi(\cdot|S_t)}[q_{\pi}(S_t,A_t)\ln\nabla\pi_{\theta}(A_t|S_t)] 
  $$</span>.</p>
  <p>Also can I assume that I can use <span class="math-container">$A_t$</span> instead of <span class="math-container">$A$</span> in all my theorems/definitions - thus adding the depoendence of <span class="math-container">$t$</span>? Next question is how terms involving rewards are written down, for example this is from the TRO paper, but for me reward is a random variable actually, while they directly define it as a function:</p>
  <p><span class="math-container">$$
  \eta(\pi) = \mathbb{E}{s_0,a_0,...}\left[\sum_{t=0}^{\infty} \gamma^t r(s_t)\right], \text{ where}
  s_0 \sim \rho_0(s_0), \quad a_t \sim \pi(a_t|s_t), \quad s_{t+1} \sim P(s_{t+1}|s_t,a_t).
  $$</span></p>
  <p>Following th conventions of the book I am following this is actually the term</p>
  <p><span class="math-container">$$
  \mathbb{E}_{S_t\sim d^\pi, A_t\sim\pi(\cdot|S_t),R_{t+1}\sim R(A_t, S_t)}[\sum_t \gamma^tR_{t+1}(S_t, A_t)] = \sum_{r_{t+1}\in R} r_{t+1}p(r_{t+1}|s_t,a_t)\sum_{s_t \in S} d^\pi(s_t)\sum_{a_t \in A}\pi(a_t,s_t)\sum_{t}\gamma^tr_{t+1}(s_t,a_t)
  $$</span></p>
  <p>isn't it?</p>
  <p>Also I am often confused about using conditional expectations vs unconditional expectations in RL litereature. Sometimes I feel like it is used interchangably - is there a reason for that? But maybe I am overthinking a lot of this. Can I show mathematically that all of those notations that I presented are really the same?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-08-30T19:05:21.737Z
Comments.Text: This is what made me give up on RL, just too many unexplained notations.
Comments.UserDisplayName: DollarAkshay
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-08-31T06:29:37.437Z
Comments.Text: yeah its a bit of a shame
Comments.UserDisplayName: craaaft
Tags: <reinforcement-learning><notation>
Title: Questions about notation in RL
UserDisplayName: craaaft
ThreadText: # Questions about notation in RL
  
  (Asked by: username_0 on 2024-08-30)
  
  I have some questions on notation in reinforcement learning, since it seems that different papers, books have some different notation. I am writing a thesis and wanted to have the notation consistent. I base my notationon this [book](https://github.com/MathFoundationRL/Book-Mathematical-Foundation-of-Reinforcement-Learning), but then a paper like TRPO has some different kind of notation. My first question is the following - some people write $\pi$ under the expectation which means that they sample the trajectory from the policy
  
  $$
  \nabla J(\theta) =\mathbb{E}\_{\pi}[\sum\_{a \in A} \pi(a|S\_t)q\_{\pi}(S\_t,a)\ln\nabla\pi\_{\theta}(a|S\_t)] = \mathbb{E}\_{\pi}[q\_{\pi}(S\_t,A\_t)\ln\nabla\pi\_{\theta}(A\_t|S\_t)]
  $$
  
  is this the same as the following though, first sampling from the stationary distribution we obtain after running the policy for some long time:
  
  $$
  \nabla J(\theta) = \mathbb{E}\_{S\_t\sim d^\pi, A\_t\sim\pi(\cdot|S\_t)}[q\_{\pi}(S\_t,A\_t)\ln\nabla\pi\_{\theta}(A\_t|S\_t)]
  $$.
  
  Also can I assume that I can use $A\_t$ instead of $A$ in all my theorems/definitions - thus adding the depoendence of $t$? Next question is how terms involving rewards are written down, for example this is from the TRO paper, but for me reward is a random variable actually, while they directly define it as a function:
  
  $$
  \eta(\pi) = \mathbb{E}{s\_0,a\_0,...}\left[\sum\_{t=0}^{\infty} \gamma^t r(s\_t)\right], \text{ where}
  s\_0 \sim \rho\_0(s\_0), \quad a\_t \sim \pi(a\_t|s\_t), \quad s\_{t+1} \sim P(s\_{t+1}|s\_t,a\_t).
  $$
  
  Following th conventions of the book I am following this is actually the term
  
  $$
  \mathbb{E}\_{S\_t\sim d^\pi, A\_t\sim\pi(\cdot|S\_t),R\_{t+1}\sim R(A\_t, S\_t)}[\sum\_t \gamma^tR\_{t+1}(S\_t, A\_t)] = \sum\_{r\_{t+1}\in R} r\_{t+1}p(r\_{t+1}|s\_t,a\_t)\sum\_{s\_t \in S} d^\pi(s\_t)\sum\_{a\_t \in A}\pi(a\_t,s\_t)\sum\_{t}\gamma^tr\_{t+1}(s\_t,a\_t)
  $$
  
  isn't it?
  
  Also I am often confused about using conditional expectations vs unconditional expectations in RL litereature. Sometimes I feel like it is used interchangably - is there a reason for that? But maybe I am overthinking a lot of this. Can I show mathematically that all of those notations that I presented are really the same?
  
  **Tags:** reinforcement-learning, notation
  > **Comments:**
  > **username_1**: This is what made me give up on RL, just too many unexplained notations.  
  > **username_0**: yeah its a bit of a shame  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-08-30. Score: 2 | Accepted answer
  
  Yes, $S\_t \sim d^\pi$ is a nice way of saying that the states are distributed according to the state distribution induced by following $\pi$. Whilst $\pi$ does not directly choose the next state, choosing the action (which is sampled directly from $\pi$) is clearly influencing what the next state will be, so the distribution of visited states will depend upon $\pi$.
  
  Generally you can just add a $t$ subscript but that would be on you to make sure it makes sense. Generally it is omitted to avoid overloading notation.
  
  For the reward, I would first point out that a random variable is a function. But you can write this however is easier for you -- generally there are different ways of writing this, e.g. $r\_t, r(s\_t, a\_t), r(s\_t, a\_t, s\_{t+1})$. The first is probably the most commonly used as it is the cleanest to read but they are all equivalent (e.g. the second term is typically used in place of the final term when the environment dynamics are deterministic and thus $s\_{t+1}$ doesn't affect the reward).
  
  > **Comments:**
  > **username_0**: Thank you! Ok so notation in regards of the policy gradient would be the same in both as I understand. True, the RV is a function but then the reward is a function of two random variables S and A. Anyhow, thank you! I am probably overthinking all of this. I was just wondering since I couldn't find any rssource that was introducing the subject using a stochastic reward.  
  > **username_0**: And do you have maybe any comments about my last question about conditional vs unconditional expectation? Conditional expectation is always when we want to specify some starting point as I assume, but often underneath the expectation sign there is indicaiton of sampling from the policy? But generally I think there should be?  
  > **username_2**: you used conditional expectation when you want to assume the outcome of an RV is known. For instance, the value function is the value of the expected returns starting from state $s$, i.e. _conditioning_ that $S_t = s$. I would expect that most authors include the conditioning in the expectation tbh, I've not really come across many examples where it is omitted when it is indeed being conditioned on.  
  > **username_0**: But given $S_t = s$ we have to further sample trajectories to determine the value function. Wouldn't it then make sense to write the sampling from a policy under the expectation sign additionally to conditioning it on $S_t = s$?  
  > **username_0**: So the two ways to state the policy gradient above are equivalent right?  
  > **username_2**: you can write this yes. e.g. you might write $v_\pi(s) = \mathbb{E}_{\tau \sim (\pi, p)} \left[ ... \right]$ where $\tau$ is the trajectory and $p$ is the environment dynamics (so we are saying the trajectory is generated according to the policy and dynamics), and the $...$ would be your usual returns and conditioning.  
  > **username_2**: the two PG's look equivalent. the second is just more explicitly saying which distribution the states are sampled from, which is the state distribution induced by the policy. generally, though, this is implied rather than explicitly stated.  
  
  ---
  
  