Answers.Body: <p>A rollout buffer is, generally speaking, a buffer for a much shorter time horizon than a replay buffer, and is discarded after use. The replay buffer in off-policy methods like DQN store all historical transitions up to some limit (usually 1 million transitions).</p>
  <p>The trajectory sampling being parallelised does help with speed of collection but it also offers better exploration and the rollout buffer is more varied -- this is especially true if you have a policy that has high entropy when taking actions and an environment with e.g. a stochastic start state.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-23T19:00:17.013Z
Answers.LastActivityDate: 2024-07-23T19:00:17.013Z
Answers.UserDisplayName: David
Body: <ol>
  <li>What is the difference between a rollout buffer and a replay buffer (as used in DQNs). Why can't they be used interchangeably?</li>
  <li>Why is the trajectory sampling parallelized? Is it just for making data collection faster?</li>
  <li>Why don't we bootstrap at each step of trajectory like TD(0) and only do it at the end of the rollout buffer ?</li>
  <li>How does the critic and actor network get updated?</li>
  </ol>
  <p>I tried reading the paper but it's a bit confusing.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-23T19:00:37.197Z
Comments.Text: could you elaborate on point 3 further please, and for point 4 do you want to know what losses are minimised for the actor and critic?
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-25T08:47:47.947Z
Comments.Text: Please, ask only one question per post, even if the questions are related to the same topic. You can always create multiple posts with some context.
Comments.UserDisplayName: nbro
Tags: <policy-gradients><actor-critic-methods><proximal-policy-optimization>
Title: I have a few doubts understanding and implementing Proximal Policy Optimisation Algorithm
UserDisplayName: DeadAsDuck
ThreadText: # I have a few doubts understanding and implementing Proximal Policy Optimisation Algorithm
  
  (Asked by: username_0 on 2024-07-23)
  
  1. What is the difference between a rollout buffer and a replay buffer (as used in DQNs). Why can't they be used interchangeably?
  2. Why is the trajectory sampling parallelized? Is it just for making data collection faster?
  3. Why don't we bootstrap at each step of trajectory like TD(0) and only do it at the end of the rollout buffer ?
  4. How does the critic and actor network get updated?
  
  I tried reading the paper but it's a bit confusing.
  
  **Tags:** policy-gradients, actor-critic-methods, proximal-policy-optimization
  > **Comments:**
  > **username_1**: could you elaborate on point 3 further please, and for point 4 do you want to know what losses are minimised for the actor and critic?  
  > **username_2**: Please, ask only one question per post, even if the questions are related to the same topic. You can always create multiple posts with some context.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-07-23. Score: 1 | Accepted answer
  
  A rollout buffer is, generally speaking, a buffer for a much shorter time horizon than a replay buffer, and is discarded after use. The replay buffer in off-policy methods like DQN store all historical transitions up to some limit (usually 1 million transitions).
  
  The trajectory sampling being parallelised does help with speed of collection but it also offers better exploration and the rollout buffer is more varied -- this is especially true if you have a policy that has high entropy when taking actions and an environment with e.g. a stochastic start state.
  
  
  ---
  
  