Answers.Body: <p>If I understand your question correctly, you are wondering whether the
  policy gradient objective coincides with some real measure of progress.
  This is exactly what the Policy Gradient Theorem proves (see Sutton et
  al. (2000) or Sutton and Barto (2018), chapter 13). In particular,
  policy gradient methods optimize the value of the start state <span class="math-container">$s_0$</span> under the
  current policy, <span class="math-container">$v_\pi(s_0)$</span>. Since this value is defined as an
  expectation over returns, then your conclusion is correct.</p>
  
  <p>Sutton, Richard S., and Andrew G. Barto. 2018. <em>Reinforcement Learning -
  an Introduction</em>. Adaptive Computation and Machine Learning. MIT Press.
  <a href="http://www.worldcat.org/oclc/37293240" rel="nofollow noreferrer">http://www.worldcat.org/oclc/37293240</a>.</p>
  
  <p>Sutton, Richard S, David A McAllester, Satinder P Singh, and Yishay
  Mansour. 2000. “Policy Gradient Methods for Reinforcement Learning with
  Function Approximation.” In <em>Advances in Neural Information Processing
  Systems</em>, 1057–63.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-26T15:21:05.750Z
Answers.Text: Ah, fantastic. I'll take a look at the proof. Thanks for the references!
Answers.UserDisplayName: lanf
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-25T12:40:56.323Z
Answers.LastActivityDate: 2020-02-25T20:50:11.643Z
Answers.LastEditDate: 2020-02-25T20:50:11.643Z
Answers.UserDisplayName: Alex L
Body: <p>Let</p>
  
  <p><span class="math-container">$$
  \nabla_\theta J(\pi_\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \left[ \sum_{t = 0}^T \nabla_\theta \log \pi_\theta (a_t|s_t) R(\tau) \right]
  $$</span>
  be the expanded expression for a simple policy gradient, where <span class="math-container">$\theta$</span> are the parameters of the policy <span class="math-container">$\pi$</span>, <span class="math-container">$J$</span> denotes the expected return function, <span class="math-container">$\tau$</span> is a trajectory of states and actions, <span class="math-container">$t$</span> is a timestep index, and <span class="math-container">$R$</span> gives the sum of rewards for a trajectory.</p>
  
  <p>Let <span class="math-container">$\mathcal{D}$</span> be the set of all trajectories used for training. An estimator of the above policy gradient is given by</p>
  
  <p><span class="math-container">$$
  \hat{g} = \frac{1}{\mathcal{D}} \sum_{\tau \in \mathcal{D}} \sum_{t = 0}^T \nabla_\theta \log \pi_\theta (a_t|s_t) R(\tau).
  $$</span>
  A loss function associated with this estimator, given a single trajectory with <span class="math-container">$T$</span> timesteps, is given by
  <span class="math-container">$$
  L(\tau) = -\sum_{t = 0}^T \log \pi_\theta (a_t|s_t) R(\tau).
  $$</span>
  Minimizing <span class="math-container">$L(\tau)$</span> by SGD or a similar algorithm will result in a working policy gradient implementation.</p>
  
  <p>My question is what is the proper terminology for this loss function? Is it an (unbiased?) estimator for the expected returns <span class="math-container">$J(\pi_\theta)$</span> if summed over all trajectories? If someone is able to provide a proof that minimizing <span class="math-container">$L$</span> maximizes <span class="math-container">$J(\pi_\theta)$</span>, or point me to a reference for this, that would be greatly appreciated.</p>
  
Tags: <reinforcement-learning><policy-gradients>
Title: Is the negative of the policy loss function in a simple policy gradient algorithm an estimator of expected returns?
UserDisplayName: lanf
ThreadText: # Is the negative of the policy loss function in a simple policy gradient algorithm an estimator of expected returns?
  
  (Asked by: username_0 on 2020-02-24)
  
  Let
  
  $$
  \nabla\_\theta J(\pi\_\theta) = \mathbb{E}\_{\tau \sim \pi\_\theta} \left[ \sum\_{t = 0}^T \nabla\_\theta \log \pi\_\theta (a\_t|s\_t) R(\tau) \right]
  $$
  be the expanded expression for a simple policy gradient, where $\theta$ are the parameters of the policy $\pi$, $J$ denotes the expected return function, $\tau$ is a trajectory of states and actions, $t$ is a timestep index, and $R$ gives the sum of rewards for a trajectory.
  
  Let $\mathcal{D}$ be the set of all trajectories used for training. An estimator of the above policy gradient is given by
  
  $$
  \hat{g} = \frac{1}{\mathcal{D}} \sum\_{\tau \in \mathcal{D}} \sum\_{t = 0}^T \nabla\_\theta \log \pi\_\theta (a\_t|s\_t) R(\tau).
  $$
  A loss function associated with this estimator, given a single trajectory with $T$ timesteps, is given by
  $$
  L(\tau) = -\sum\_{t = 0}^T \log \pi\_\theta (a\_t|s\_t) R(\tau).
  $$
  Minimizing $L(\tau)$ by SGD or a similar algorithm will result in a working policy gradient implementation.
  
  My question is what is the proper terminology for this loss function? Is it an (unbiased?) estimator for the expected returns $J(\pi\_\theta)$ if summed over all trajectories? If someone is able to provide a proof that minimizing $L$ maximizes $J(\pi\_\theta)$, or point me to a reference for this, that would be greatly appreciated.
  
  **Tags:** reinforcement-learning, policy-gradients
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-02-25. Score: 2 | Accepted answer
  
  If I understand your question correctly, you are wondering whether the
  policy gradient objective coincides with some real measure of progress.
  This is exactly what the Policy Gradient Theorem proves (see Sutton et
  al. (2000) or Sutton and Barto (2018), chapter 13). In particular,
  policy gradient methods optimize the value of the start state $s\_0$ under the
  current policy, $v\_\pi(s\_0)$. Since this value is defined as an
  expectation over returns, then your conclusion is correct.
  
  Sutton, Richard S., and Andrew G. Barto. 2018. *Reinforcement Learning -
  an Introduction*. Adaptive Computation and Machine Learning. MIT Press.
  <http://www.worldcat.org/oclc/37293240>.
  
  Sutton, Richard S, David A McAllester, Satinder P Singh, and Yishay
  Mansour. 2000. “Policy Gradient Methods for Reinforcement Learning with
  Function Approximation.” In *Advances in Neural Information Processing
  Systems*, 1057–63.
  
  > **Comments:**
  > **username_0**: Ah, fantastic. I'll take a look at the proof. Thanks for the references!  
  
  ---
  
  