Answers.Body: <p>We do decorrelate training experience, even for policy gradient methods. This is because decorrelation helps training data be more like IID data, which helps with the convergence of SGD-like optimizers. </p>
  
  <p>The shuffling is done on line 151 of OpenAI's <a href="https://github.com/openai/baselines/blob/master/baselines/ppo2/ppo2.py#L151" rel="nofollow noreferrer">"baselines" implementation of PPO</a>.</p>
  
  <p>I'm going to guess that there's a bug somewhere in your implementation. If you're using the same truncated advantage estimation as in the <a href="https://arxiv.org/pdf/1707.06347.pdf" rel="nofollow noreferrer">PPO paper</a>, </p>
  
  <p><span class="math-container">$$\begin{align}
  &amp;\hat{A}_t = \delta_t + (\gamma\lambda)\delta_{t+1}+\dots+(\gamma\lambda)^{T-t+1}\delta_{T-1}\\
  &amp;\text{where}\quad\delta_t=r_t+\gamma V(s_{t+1})-V(s_t)
  \end{align}$$</span></p>
  
  <p>make sure you're not shuffling experience before computing your advantage estimates <span class="math-container">$\hat{A}_1,\dots,\hat{A}_T$</span>. Each of these estimates is a function of several unbroken steps of experience. After you compute your advantage estimates, then create the tuples <span class="math-container">$(s_t, a_t, \hat{A}_t)$</span> and shuffle and sample from those. I think that's all the information you need  per time step for constructing their objective function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-17T09:56:09.450Z
Answers.LastActivityDate: 2019-03-17T10:10:38.300Z
Answers.LastEditDate: 2019-03-17T10:10:38.300Z
Answers.UserDisplayName: Philip Raeisghasem
Body: <p>I'm implementing PPO myself strictly follow the steps:</p>
  
  <ol>
  <li>sample transitions</li>
  <li>randomly shuffle the sampled transitions</li>
  <li>compute gradients and update networks using the sampled transitions</li>
  <li>drop transitions and repeat the above steps </li>
  </ol>
  
  <p>I observe a strange phenomenon that randomly shuffling transitions makes the algorithm perform significantly worse than keeping it as it is. This is very strange to me. To my best understanding, neural networks perform badly when the input data are correlated. To decorrelate transitions, algorithms like DQN introduce replay buffer and randomly sample from it. But this seems not the same story to policy-based methods. I'm wondering why policy-based methods do not require to decorrelate the input data?</p>
  
Tags: <reinforcement-learning><proximal-policy-optimization>
Title: Why don't we decorrelate transitions for policy-based data?
UserDisplayName: Maybe
ThreadText: # Why don't we decorrelate transitions for policy-based data?
  
  (Asked by: username_0 on 2019-03-17)
  
  I'm implementing PPO myself strictly follow the steps:
  
  1. sample transitions
  2. randomly shuffle the sampled transitions
  3. compute gradients and update networks using the sampled transitions
  4. drop transitions and repeat the above steps
  
  I observe a strange phenomenon that randomly shuffling transitions makes the algorithm perform significantly worse than keeping it as it is. This is very strange to me. To my best understanding, neural networks perform badly when the input data are correlated. To decorrelate transitions, algorithms like DQN introduce replay buffer and randomly sample from it. But this seems not the same story to policy-based methods. I'm wondering why policy-based methods do not require to decorrelate the input data?
  
  **Tags:** reinforcement-learning, proximal-policy-optimization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-03-17. Score: 4 | Accepted answer
  
  We do decorrelate training experience, even for policy gradient methods. This is because decorrelation helps training data be more like IID data, which helps with the convergence of SGD-like optimizers.
  
  The shuffling is done on line 151 of OpenAI's ["baselines" implementation of PPO](https://github.com/openai/baselines/blob/master/baselines/ppo2/ppo2.py#L151).
  
  I'm going to guess that there's a bug somewhere in your implementation. If you're using the same truncated advantage estimation as in the [PPO paper](https://arxiv.org/pdf/1707.06347.pdf),
  
  $$\begin{align}
  &\hat{A}\_t = \delta\_t + (\gamma\lambda)\delta\_{t+1}+\dots+(\gamma\lambda)^{T-t+1}\delta\_{T-1}\\
  &\text{where}\quad\delta\_t=r\_t+\gamma V(s\_{t+1})-V(s\_t)
  \end{align}$$
  
  make sure you're not shuffling experience before computing your advantage estimates $\hat{A}\_1,\dots,\hat{A}\_T$. Each of these estimates is a function of several unbroken steps of experience. After you compute your advantage estimates, then create the tuples $(s\_t, a\_t, \hat{A}\_t)$ and shuffle and sample from those. I think that's all the information you need per time step for constructing their objective function.
  
  
  ---
  
  