Answers.Body: <p>In practice policy optimization algorithms like TRPO collect trajectories of experiences by interacting with the environment under the current policy, thus the sloppy notation here with expectation over <span class="math-container">$t$</span> just signifies that the empirical estimate of the surrogate objective is taken over all time steps in the collected trajectories. Including just <span class="math-container">$t$</span> under the expectation is a simpler and more compact though sloppy and unconventional way to convey this, while your proposal is usually computationally efficient to sample local state-action pairs from a batch of collected experiences replay buffer rather than needing to consider entire trajectories such as DQN, PPO, A2C, etc.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-08T05:49:31.977Z
Answers.LastActivityDate: 2024-09-08T06:06:32.027Z
Answers.LastEditDate: 2024-09-08T06:06:32.027Z
Answers.UserDisplayName: cinch
Body: <p>Slide 11, <a href="https://cs.uwaterloo.ca/%7Eppoupart/teaching/cs885-spring18/slides/cs885-lecture15b.pdf" rel="nofollow noreferrer">https://cs.uwaterloo.ca/~ppoupart/teaching/cs885-spring18/slides/cs885-lecture15b.pdf</a></p>
  <p><a href="https://i.sstatic.net/IYdUjboW.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/IYdUjboW.png" alt="enter image description here" /></a></p>
  <p>Why is <span class="math-container">$t$</span> included under the expectation? Normally, instead of &quot;t&quot;, I would expect <span class="math-container">$(a_t,s_t) \sim \pi_{\theta_{old}} $</span>.</p>
  
Tags: <reinforcement-learning><trust-region-policy-optimization>
Title: Unclear point in TRPO
UserDisplayName: DSPinfinity
ThreadText: # Unclear point in TRPO
  
  (Asked by: username_0 on 2024-09-03)
  
  Slide 11, [https://cs.uwaterloo.ca/~ppoupart/teaching/cs885-spring18/slides/cs885-lecture15b.pdf](https://cs.uwaterloo.ca/%7Eppoupart/teaching/cs885-spring18/slides/cs885-lecture15b.pdf)
  
  [![enter image description here](https://i.sstatic.net/IYdUjboW.png)](https://i.sstatic.net/IYdUjboW.png)
  
  Why is $t$ included under the expectation? Normally, instead of "t", I would expect $(a\_t,s\_t) \sim \pi\_{\theta\_{old}} $.
  
  **Tags:** reinforcement-learning, trust-region-policy-optimization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-09-08. Score: 1 | Accepted answer
  
  In practice policy optimization algorithms like TRPO collect trajectories of experiences by interacting with the environment under the current policy, thus the sloppy notation here with expectation over $t$ just signifies that the empirical estimate of the surrogate objective is taken over all time steps in the collected trajectories. Including just $t$ under the expectation is a simpler and more compact though sloppy and unconventional way to convey this, while your proposal is usually computationally efficient to sample local state-action pairs from a batch of collected experiences replay buffer rather than needing to consider entire trajectories such as DQN, PPO, A2C, etc.
  
  
  ---
  
  