Answers.Body: <p>I guess the gradient of the expectation of the Utility function, <span class="math-container">$\nabla_{\theta}J(\theta)$</span> in policy gradient methods where <span class="math-container">$\nabla_{\theta}J(\theta) = E_{\tau \sim p(\tau ; \theta)}[r(\tau)\nabla_{\theta}log p(\tau;\theta)]$</span> can be approximated using a single sample trajectory as shown in a <a href="https://www.youtube.com/watchv=lvoHnicueoE&amp;list=RDCMUCdKG2JnvPu6mY1NDXYFfN0g&amp;index=1" rel="nofollow noreferrer">deep reinforcement learning lecture by stanford</a> where <span class="math-container">$J(\theta) \approx \sum_{t &gt; 0}r(\tau)\nabla_{\theta}log\pi_{\theta}(a_t |s_t)$</span> and an average over sampled trajectories is not needed to compute the gradient for <span class="math-container">$\theta$</span> update in the direction of gradient.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-13T15:14:32.627Z
Answers.LastActivityDate: 2020-05-13T15:14:32.627Z
Answers.UserDisplayName: calveeen
Body: <p>The <a href="https://www.youtube.com/watch?v=S_gwYj1Q-44&amp;list=PLAdk-EyP1ND8MqJEJnSvaoUShrAWYe51U&amp;index=4" rel="nofollow noreferrer">Deep RL bootcamp on policy gradient techniques</a> gives the update equation for the policy network in A3C as</p>
  
  <p><span class="math-container">$\theta_{i+1} = \theta_i + \alpha \times 1/m \sum_{k=1}^m\sum_{t=0}^{H-1}\nabla_{\theta}log\pi_{\theta_i}(u_t^{(k)} | s_t^{(k)})(Q(s_t^{(k)},u_t^{(k)}) - V_{\Phi_i}^\pi(s_t^{(k)}))    $</span></p>
  
  <p>However in the <a href="https://arxiv.org/abs/1602.01783" rel="nofollow noreferrer">actual A3C paper</a>, the gradient update is based on a single trajectory and there is no averaging of the gradient over <span class="math-container">$m$</span> trajectories as defined in the video ? The simple <a href="https://lilianweng.github.io/lil-log/2018/04/08/policy-gradient-algorithms.html#a2c" rel="nofollow noreferrer">action-value actor-critic algorithm</a> also does not seem to require an averaging over m trajectory. </p>
  
Tags: <policy-gradients><actor-critic-methods>
Title: Why a single trajectory can be used to update the policy network $\theta$ in A3C?
UserDisplayName: calveeen
ThreadText: # Why a single trajectory can be used to update the policy network $\theta$ in A3C?
  
  (Asked by: username_0 on 2020-05-12)
  
  The [Deep RL bootcamp on policy gradient techniques](https://www.youtube.com/watch?v=S_gwYj1Q-44&list=PLAdk-EyP1ND8MqJEJnSvaoUShrAWYe51U&index=4) gives the update equation for the policy network in A3C as
  
  $\theta\_{i+1} = \theta\_i + \alpha \times 1/m \sum\_{k=1}^m\sum\_{t=0}^{H-1}\nabla\_{\theta}log\pi\_{\theta\_i}(u\_t^{(k)} | s\_t^{(k)})(Q(s\_t^{(k)},u\_t^{(k)}) - V\_{\Phi\_i}^\pi(s\_t^{(k)})) $
  
  However in the [actual A3C paper](https://arxiv.org/abs/1602.01783), the gradient update is based on a single trajectory and there is no averaging of the gradient over $m$ trajectories as defined in the video ? The simple [action-value actor-critic algorithm](https://lilianweng.github.io/lil-log/2018/04/08/policy-gradient-algorithms.html#a2c) also does not seem to require an averaging over m trajectory.
  
  **Tags:** policy-gradients, actor-critic-methods
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-05-13. Score: 1
  
  I guess the gradient of the expectation of the Utility function, $\nabla\_{\theta}J(\theta)$ in policy gradient methods where $\nabla\_{\theta}J(\theta) = E\_{\tau \sim p(\tau ; \theta)}[r(\tau)\nabla\_{\theta}log p(\tau;\theta)]$ can be approximated using a single sample trajectory as shown in a [deep reinforcement learning lecture by stanford](https://www.youtube.com/watchv=lvoHnicueoE&list=RDCMUCdKG2JnvPu6mY1NDXYFfN0g&index=1) where $J(\theta) \approx \sum\_{t > 0}r(\tau)\nabla\_{\theta}log\pi\_{\theta}(a\_t |s\_t)$ and an average over sampled trajectories is not needed to compute the gradient for $\theta$ update in the direction of gradient.
  
  
  ---
  
  