Answers.Body: <p>First, the derivative is usually taken with respect to a variable (input) of the function. Hence the notation <span class="math-container">$\frac{df}{dx}$</span> for some function <span class="math-container">$f(x)$</span>.</p>
  
  <p>If you look at your equation more carefully</p>
  
  <p><span class="math-container">$$\nabla log P(\tau^{i};\theta) = \Sigma_{t=0}\nabla_{\theta}log\pi(a_{t}|s_t, \theta).$$</span></p>
  
  <p>You will see that the gradient is taken with respect to <span class="math-container">$\theta$</span>, which are the parameters (i.e. a vector) e.g. of your neural network, that is, <span class="math-container">$\nabla_{\theta}$</span>. </p>
  
  <p>In this case, it doesn't really matter whether <span class="math-container">$\pi$</span> represents a distribution or not (for some specific value of <span class="math-container">$\theta$</span>), but you're right that <span class="math-container">$\pi$</span> often represents a probability distribution over the possible actions (given a specific state). In any case, <span class="math-container">$\pi$</span> is a function of the parameters <span class="math-container">$\theta$</span> (i.e. in the case of a distribution, <span class="math-container">$\pi_{\theta}$</span> is a family of distributions for all possible values of <span class="math-container">$\theta$</span>), i.e. if you change <span class="math-container">$\theta$</span> the outputs of <span class="math-container">$\pi$</span> also change, so you can take the derivative of it with respect to <span class="math-container">$\theta$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-21T18:15:04.813Z
Answers.LastActivityDate: 2020-02-21T18:29:19.887Z
Answers.LastEditDate: 2020-02-21T18:29:19.887Z
Answers.UserDisplayName: nbro
Body: <p>I am learning about policy gradient methods from the Deep RL Bootcamp by Peter Abbeel and I am a bit stumbled by the math presented. In the lecture, he derives the gradient logarithm likelihood of a trajectory to be</p>
  
  <p><span class="math-container">$$\nabla log P(\tau^{i};\theta) = \Sigma_{t=0}\nabla_{\theta}log\pi(a_{t}|s_t, \theta).$$</span></p>
  
  <p>Is <span class="math-container">$\pi(a_{t} | s_{t}, \theta)$</span> a distribution or a function? Because a derivative can only be taken wrt a function. My understanding is that <span class="math-container">$\pi(a_{t},s_{t}, \theta)$</span> is usually represented as a distribution of actions over states, since input of a neural network for policy gradient would be the <span class="math-container">$s_t$</span> and output would be <span class="math-container">$\pi(a_t, s_t)$</span>, using model weights <span class="math-container">$\theta$</span>.</p>
  
Tags: <math><deep-rl><policy-gradients><stochastic-policy>
Title: In the policy gradient equation, is $\pi(a_{t} | s_{t}, \theta)$ a distribution or a function?
UserDisplayName: calveeen
ThreadText: # In the policy gradient equation, is $\pi(a_{t} | s_{t}, \theta)$ a distribution or a function?
  
  (Asked by: username_0 on 2020-02-21)
  
  I am learning about policy gradient methods from the Deep RL Bootcamp by Peter Abbeel and I am a bit stumbled by the math presented. In the lecture, he derives the gradient logarithm likelihood of a trajectory to be
  
  $$\nabla log P(\tau^{i};\theta) = \Sigma\_{t=0}\nabla\_{\theta}log\pi(a\_{t}|s\_t, \theta).$$
  
  Is $\pi(a\_{t} | s\_{t}, \theta)$ a distribution or a function? Because a derivative can only be taken wrt a function. My understanding is that $\pi(a\_{t},s\_{t}, \theta)$ is usually represented as a distribution of actions over states, since input of a neural network for policy gradient would be the $s\_t$ and output would be $\pi(a\_t, s\_t)$, using model weights $\theta$.
  
  **Tags:** math, deep-rl, policy-gradients, stochastic-policy
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-02-21. Score: 4 | Accepted answer
  
  First, the derivative is usually taken with respect to a variable (input) of the function. Hence the notation $\frac{df}{dx}$ for some function $f(x)$.
  
  If you look at your equation more carefully
  
  $$\nabla log P(\tau^{i};\theta) = \Sigma\_{t=0}\nabla\_{\theta}log\pi(a\_{t}|s\_t, \theta).$$
  
  You will see that the gradient is taken with respect to $\theta$, which are the parameters (i.e. a vector) e.g. of your neural network, that is, $\nabla\_{\theta}$.
  
  In this case, it doesn't really matter whether $\pi$ represents a distribution or not (for some specific value of $\theta$), but you're right that $\pi$ often represents a probability distribution over the possible actions (given a specific state). In any case, $\pi$ is a function of the parameters $\theta$ (i.e. in the case of a distribution, $\pi\_{\theta}$ is a family of distributions for all possible values of $\theta$), i.e. if you change $\theta$ the outputs of $\pi$ also change, so you can take the derivative of it with respect to $\theta$.
  
  
  ---
  
  