Answers.Body: <p>From the equation you wrote:</p>
  <ul>
  <li><span class="math-container">$U(\theta)$</span> is the <strong>total expected reward</strong> of a policy <span class="math-container">$\pi_\theta$</span> (with parameters/weights <span class="math-container">$\theta$</span>) over <em>all</em> possible trajectories <span class="math-container">$\tau$</span> sampled under the environment dynamics while following the policy itself: the term <span class="math-container">$P(\tau;\theta)$</span> should include both the env. dynamics (i.e. how it evolves each step given an action from a state), usually denoted as <span class="math-container">$P(s'\mid s, a)$</span> where <span class="math-container">$s'$</span> is next state, and the fact that we sample actions from the policy <span class="math-container">$a\sim \pi_\theta(s)$</span> from a state <span class="math-container">$s$</span>.</li>
  <li>For practical purpose, you can't take the expectation over all possible trajectories so you run the policy over <span class="math-container">$m$</span> of them and approximate the expectation (<em>integral</em>) with a summation over finite num. of trajectories. Thus, <span class="math-container">$\hat g = \frac{1}{m}\sum_{i=1}^m\nabla_\theta \log P(\tau^{(i)};\theta)R(\tau^{(i)})$</span>.</li>
  </ul>
  <blockquote>
  <p>My question is how does this work?</p>
  </blockquote>
  <p>Basically the formula that estimates the gradient <span class="math-container">$\hat g$</span> makes use of what is called a <em>score estimator</em> (also called REINFORCE from the notable algorithm). It works by computing the probability of a trajectory to occur, then you score that trajectory by <span class="math-container">$R(\tau^{(i)})$</span> the <em>total (discounted) reward</em>, and finally compute the expectation as a simple average over <span class="math-container">$m$</span> trajectories.</p>
  <p>The gradient of the policy <span class="math-container">$\nabla_\theta$</span> is so scored by the log-prob times the reward: the log-prob term has the intuitive explanation of increasing the probability of trajectories that score a high <span class="math-container">$R(\tau)$</span>, while decreasing the ones for which <span class="math-container">$R$</span> is bad.</p>
  <blockquote>
  <p>doesn't taking multiple trials give you a more accurate guess of the point <span class="math-container">$(\theta,U(\theta))$</span> alone?</p>
  </blockquote>
  <p>Increasing <span class="math-container">$m$</span> provides a better estimate of <span class="math-container">$\hat g$</span>. Indeed, as <span class="math-container">$m\to\infty$</span> you recover the true gradient <span class="math-container">$g$</span> by computing the full expectation. But that is not feasible in practice.</p>
  <blockquote>
  <p>I don't get how you can get the gradient without evaluating the function at different <span class="math-container">$\theta$</span>'s</p>
  </blockquote>
  <p>At each update/optimization step the policy is only evaluated at <span class="math-container">$\theta$</span>: its weights define both the point at which the gradient is computed and the policy itself, since it's a parametrized function approximator (like a neural network.) So, there is no need to know/evaluate at the surroundings of <span class="math-container">$\theta$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-25T16:04:33.650Z
Answers.Text: So is the ground truth of $R(\tau)$ fixed? The only thing that changes when varying $\theta$ is the probability of following that trajectory $\tau$?
Answers.UserDisplayName: User
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-25T19:47:17.673Z
Answers.Text: Exactly. For a given trajectory $\tau$, $R(\tau)$ is always the same since it's just a (discounted) sum over the immediate rewards $r_t$ gathered in $\tau$: it depends on the environment not the agent/policy.  Instead, the log-prob changes as the parameters $\theta$ got updated.
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-26T03:30:41.480Z
Answers.Text: Sorry, I have just one more question, isn't the $U(\theta)$ function defined in essentially the way that the value function is (i.e. total expected reward of a policy)? Except it doesn't take in the state as an input, rather the parameters themselves?
Answers.UserDisplayName: User
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-26T08:01:31.983Z
Answers.Text: $U(\theta)$ and $V(s)$ are similar (I assume $V(s)=V^{\pi_\theta}(s)$), but not equal. Although both represent sum of rewards, $V(s)$ computes that only from a state, while $U(\theta)$ considers an initial $(s_0,a_0)$ pair: this looks like a Q-value, $Q(s,a)$, but it's still different because $U(\theta)$ also averages over the initial state distribution $s_0\sim\rho(s_0)$. Intuitively, it's like $U(\theta) = \mathbb{E}_{s_0\sim \rho}[V(s_0);\ \pi_\theta]$ (but I'm not sure if 100% correct) as after all the RL objective is to maximize the total reward *expected* over all possible trajectories.
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-25T08:41:01.130Z
Answers.LastActivityDate: 2023-04-25T08:41:01.130Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>If I understand it correctly from the following equation</p>
  <p><span class="math-container">$$U(\theta)=\mathbb{E}_{\tau \sim P(\tau;\theta)}\left [ \sum_{t=0}^{H-1}R(s_t,u_t);\pi_{\theta} \right ]=\sum_{\tau}P(\tau;\theta)R(\tau)$$</span></p>
  <p>from <a href="https://davidmeyer.github.io/ml/policy_gradient_methods_for_robotics.pdf" rel="nofollow noreferrer">this paper</a>, the utility of a policy parameterized by weights <span class="math-container">$\theta$</span>, is the total expected reward following (all, one?) trajectory <span class="math-container">$\tau$</span>. After rearranging, taking the gradient, and running the policy over <span class="math-container">$m$</span> trials, the equation becomes</p>
  <p><span class="math-container">$$\nabla_{\theta}U(\theta)\approx \hat{g} = \frac{1}{m}\sum_{i=1}^{m}\nabla_{\theta}logP(\tau^{(i)};\theta)R(\tau^{(i)}).$$</span></p>
  <p>My question is - how does this work? I can't understand it in terms of the utility/loss surface. At most, doesn't taking multiple trials give you a more accurate guess of the point <span class="math-container">$(\theta,U(\theta))$</span> alone? I don't get how you can get the gradient (a sort of local tangent plane of a surface) without evaluating the function at different <span class="math-container">$\theta$</span>'s, like how do you get the slope at a point (2,3) without evaluating it at 2.01, 1.99, etc. at the very least? Is it by virtue of the well defined geometry of the function <span class="math-container">$U$</span> we defined?</p>
  
Tags: <reinforcement-learning><backpropagation><policy-gradients><calculus>
Title: How do policy gradients work?
UserDisplayName: User
ThreadText: # How do policy gradients work?
  
  (Asked by: username_0 on 2023-04-25)
  
  If I understand it correctly from the following equation
  
  $$U(\theta)=\mathbb{E}\_{\tau \sim P(\tau;\theta)}\left [ \sum\_{t=0}^{H-1}R(s\_t,u\_t);\pi\_{\theta} \right ]=\sum\_{\tau}P(\tau;\theta)R(\tau)$$
  
  from [this paper](https://davidmeyer.github.io/ml/policy_gradient_methods_for_robotics.pdf), the utility of a policy parameterized by weights $\theta$, is the total expected reward following (all, one?) trajectory $\tau$. After rearranging, taking the gradient, and running the policy over $m$ trials, the equation becomes
  
  $$\nabla\_{\theta}U(\theta)\approx \hat{g} = \frac{1}{m}\sum\_{i=1}^{m}\nabla\_{\theta}logP(\tau^{(i)};\theta)R(\tau^{(i)}).$$
  
  My question is - how does this work? I can't understand it in terms of the utility/loss surface. At most, doesn't taking multiple trials give you a more accurate guess of the point $(\theta,U(\theta))$ alone? I don't get how you can get the gradient (a sort of local tangent plane of a surface) without evaluating the function at different $\theta$'s, like how do you get the slope at a point (2,3) without evaluating it at 2.01, 1.99, etc. at the very least? Is it by virtue of the well defined geometry of the function $U$ we defined?
  
  **Tags:** reinforcement-learning, backpropagation, policy-gradients, calculus
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-04-25. Score: 2 | Accepted answer
  
  From the equation you wrote:
  
  * $U(\theta)$ is the **total expected reward** of a policy $\pi\_\theta$ (with parameters/weights $\theta$) over *all* possible trajectories $\tau$ sampled under the environment dynamics while following the policy itself: the term $P(\tau;\theta)$ should include both the env. dynamics (i.e. how it evolves each step given an action from a state), usually denoted as $P(s'\mid s, a)$ where $s'$ is next state, and the fact that we sample actions from the policy $a\sim \pi\_\theta(s)$ from a state $s$.
  * For practical purpose, you can't take the expectation over all possible trajectories so you run the policy over $m$ of them and approximate the expectation (*integral*) with a summation over finite num. of trajectories. Thus, $\hat g = \frac{1}{m}\sum\_{i=1}^m\nabla\_\theta \log P(\tau^{(i)};\theta)R(\tau^{(i)})$.
  
  > My question is how does this work?
  
  Basically the formula that estimates the gradient $\hat g$ makes use of what is called a *score estimator* (also called REINFORCE from the notable algorithm). It works by computing the probability of a trajectory to occur, then you score that trajectory by $R(\tau^{(i)})$ the *total (discounted) reward*, and finally compute the expectation as a simple average over $m$ trajectories.
  
  The gradient of the policy $\nabla\_\theta$ is so scored by the log-prob times the reward: the log-prob term has the intuitive explanation of increasing the probability of trajectories that score a high $R(\tau)$, while decreasing the ones for which $R$ is bad.
  
  > doesn't taking multiple trials give you a more accurate guess of the point $(\theta,U(\theta))$ alone?
  
  Increasing $m$ provides a better estimate of $\hat g$. Indeed, as $m\to\infty$ you recover the true gradient $g$ by computing the full expectation. But that is not feasible in practice.
  
  > I don't get how you can get the gradient without evaluating the function at different $\theta$'s
  
  At each update/optimization step the policy is only evaluated at $\theta$: its weights define both the point at which the gradient is computed and the policy itself, since it's a parametrized function approximator (like a neural network.) So, there is no need to know/evaluate at the surroundings of $\theta$.
  
  > **Comments:**
  > **username_0**: So is the ground truth of $R(\tau)$ fixed? The only thing that changes when varying $\theta$ is the probability of following that trajectory $\tau$?  
  > **username_1**: Exactly. For a given trajectory $\tau$, $R(\tau)$ is always the same since it's just a (discounted) sum over the immediate rewards $r_t$ gathered in $\tau$: it depends on the environment not the agent/policy.  Instead, the log-prob changes as the parameters $\theta$ got updated.  
  > **username_0**: Sorry, I have just one more question, isn't the $U(\theta)$ function defined in essentially the way that the value function is (i.e. total expected reward of a policy)? Except it doesn't take in the state as an input, rather the parameters themselves?  
  > **username_1**: $U(\theta)$ and $V(s)$ are similar (I assume $V(s)=V^{\pi_\theta}(s)$), but not equal. Although both represent sum of rewards, $V(s)$ computes that only from a state, while $U(\theta)$ considers an initial $(s_0,a_0)$ pair: this looks like a Q-value, $Q(s,a)$, but it's still different because $U(\theta)$ also averages over the initial state distribution $s_0\sim\rho(s_0)$. Intuitively, it's like $U(\theta) = \mathbb{E}_{s_0\sim \rho}[V(s_0);\ \pi_\theta]$ (but I'm not sure if 100% correct) as after all the RL objective is to maximize the total reward *expected* over all possible trajectories.  
  
  ---
  
  