  For reinforcement learning on complex stochastic systems where many factors
dynamically impact the output trajectories, it is desirable to effectively
leverage the information from historical samples collected in previous
iterations to accelerate policy optimization. Classical experience replay
allows agents to remember by reusing historical observations. However, the
uniform reuse strategy that treats all observations equally overlooks the
relative importance of different samples. To overcome this limitation, we
propose a general variance reduction based experience replay (VRER) framework
that can selectively reuse the most relevant samples to improve policy gradient
estimation. This selective mechanism can adaptively put more weight on past
samples that are more likely to be generated by the current target
distribution. Our theoretical and empirical studies show that the proposed VRER
can accelerate the learning of optimal policy and enhance the performance of
state-of-the-art policy optimization approaches.

Machine Learning;Artificial Intelligence