Answers.Body: <p>Your intuition is correct as in your reference just above this equation they explain as:</p>
  <blockquote>
  <p>Notice that the right-hand side of the policy gradient theorem is a sum over states weighted by how often the states occur under the target policy <span class="math-container">$\pi$</span>; if <span class="math-container">$\pi$</span> is followed, then states will be encountered in these proportions.</p>
  </blockquote>
  <p>And the definition of the distribution <span class="math-container">$\mu(s)$</span> is also defined at the bottom of the previous page of your quoted section.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-19T12:05:11.180Z
Answers.Text: Thanks for the explanation. But what is `S_t` here and why did we substitute `s` in that expression?
Answers.UserDisplayName: DeadAsDuck
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-19T20:05:49.727Z
Answers.Text: The added subscript $t$ is to express the parameter update rule of the next equation (13.7) in the same page at any generic time step. There's exact definition linking rv $S_t$ and any generic state $s$ in the beginning of this chapter $\pi(a | s,\mathbf{\theta})=\text{Pr(}A_t =a | S_t =s, \mathbf{\theta}_t =\mathbf{\theta})$. Hope this clarifies your remaining confusion.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-19T01:10:19.637Z
Answers.LastActivityDate: 2024-07-19T01:10:19.637Z
Answers.UserDisplayName: cinch
Answers.Body: <p>In general, suppose we have a discrete random variable <span class="math-container">$X$</span>. The expectation of <span class="math-container">$X$</span> is defined as
  <span class="math-container">$$\mathbb{E}[X] = \sum_{x\in \mathcal{X}}x \times \mathbb{P}(X = x) \; ;$$</span>
  where <span class="math-container">$\mathcal{X}$</span> is the support of <span class="math-container">$X$</span>.</p>
  <p>Given that <span class="math-container">$\mu(s)$</span> is the state distribution, it is equivalent to say that
  <span class="math-container">$$\mathbb{E}\left[\sum_a q_\pi (S_t, a) \nabla \pi(a|S_t, \theta)\right] = \sum_s \mu(s) \left(\sum_a q_\pi (S_t, a) \nabla \pi(a|S_t, \theta)\right) \; ;$$</span>
  since <span class="math-container">$\mu(s)$</span> is the state distribution and <span class="math-container">$S_t$</span> is used to denote the state as a random variable.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-19T11:24:44.673Z
Answers.LastActivityDate: 2024-07-19T11:24:44.673Z
Answers.UserDisplayName: David
Body: <p>After reading Sutton and Barto, I was able to understand the derivation of this theorem. The only thing I don't get is the following part from REINFORCE algorithm:</p>
  <p><a href="https://i.sstatic.net/744SyAeK.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/744SyAeK.png" alt="enter image description here" /></a></p>
  <p>How are these terms equivalent, and how did <span class="math-container">$s$</span> get replaced with <span class="math-container">$S_t$</span>?</p>
  <p>As far as I understand, the first term is the the marginal expectation of the function</p>
  <p><a href="https://i.sstatic.net/wjJgK16Y.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/wjJgK16Y.png" alt="enter image description here" /></a></p>
  <p>over <span class="math-container">$s$</span>, and hence can be written like the second term.</p>
  <p>Also, I'm assuming the proportionality constant gets absorbed in the step size and hence we get rid of the proportionality sign.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-19T08:46:23.277Z
Comments.Text: re: the proportionality becoming an equals, you should read like this: the gradient is proportional to the RHS on the top line, and this is in turn equal to the RHS on the second line. So, the gradient is still proportional to the RHS on the second line, the equals is just meaning that it is equal to the top line RHS. To see why it is equal, just not that we are just re-writing the sum as an explicit expectation (which it is, because $\mu(s)$ is a distribution).
Comments.UserDisplayName: David
Tags: <reinforcement-learning><math><policy-gradients><sutton-barto><reinforce>
Title: How are these two terms equivalent in Sutton and Barto's derivation of the REINFORCE algorithm
UserDisplayName: DeadAsDuck
ThreadText: # How are these two terms equivalent in Sutton and Barto's derivation of the REINFORCE algorithm
  
  (Asked by: username_0 on 2024-07-19)
  
  After reading Sutton and Barto, I was able to understand the derivation of this theorem. The only thing I don't get is the following part from REINFORCE algorithm:
  
  [![enter image description here](https://i.sstatic.net/744SyAeK.png)](https://i.sstatic.net/744SyAeK.png)
  
  How are these terms equivalent, and how did $s$ get replaced with $S\_t$?
  
  As far as I understand, the first term is the the marginal expectation of the function
  
  [![enter image description here](https://i.sstatic.net/wjJgK16Y.png)](https://i.sstatic.net/wjJgK16Y.png)
  
  over $s$, and hence can be written like the second term.
  
  Also, I'm assuming the proportionality constant gets absorbed in the step size and hence we get rid of the proportionality sign.
  
  **Tags:** reinforcement-learning, math, policy-gradients, sutton-barto, reinforce
  
  > **Comments:**
  > **username_1**: re: the proportionality becoming an equals, you should read like this: the gradient is proportional to the RHS on the top line, and this is in turn equal to the RHS on the second line. So, the gradient is still proportional to the RHS on the second line, the equals is just meaning that it is equal to the top line RHS. To see why it is equal, just not that we are just re-writing the sum as an explicit expectation (which it is, because $\mu(s)$ is a distribution).  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2024-07-19. Score: 1 | Accepted answer
  
  Your intuition is correct as in your reference just above this equation they explain as:
  
  > Notice that the right-hand side of the policy gradient theorem is a sum over states weighted by how often the states occur under the target policy $\pi$; if $\pi$ is followed, then states will be encountered in these proportions.
  
  And the definition of the distribution $\mu(s)$ is also defined at the bottom of the previous page of your quoted section.
  
  > **Comments:**
  > **username_0**: Thanks for the explanation. But what is `S_t` here and why did we substitute `s` in that expression?  
  > **username_2**: The added subscript $t$ is to express the parameter update rule of the next equation (13.7) in the same page at any generic time step. There's exact definition linking rv $S_t$ and any generic state $s$ in the beginning of this chapter $\pi(a | s,\mathbf{\theta})=\text{Pr(}A_t =a | S_t =s, \mathbf{\theta}_t =\mathbf{\theta})$. Hope this clarifies your remaining confusion.  
  
  ---
  
  ### Answer by username_1 on 2024-07-19. Score: 2
  
  In general, suppose we have a discrete random variable $X$. The expectation of $X$ is defined as
  $$\mathbb{E}[X] = \sum\_{x\in \mathcal{X}}x \times \mathbb{P}(X = x) \; ;$$
  where $\mathcal{X}$ is the support of $X$.
  
  Given that $\mu(s)$ is the state distribution, it is equivalent to say that
  $$\mathbb{E}\left[\sum\_a q\_\pi (S\_t, a) \nabla \pi(a|S\_t, \theta)\right] = \sum\_s \mu(s) \left(\sum\_a q\_\pi (S\_t, a) \nabla \pi(a|S\_t, \theta)\right) \; ;$$
  since $\mu(s)$ is the state distribution and $S\_t$ is used to denote the state as a random variable.
  
  
  ---
  
  