  In reinforcement learning (RL), experience replay-based sampling techniques
play a crucial role in promoting convergence by eliminating spurious
correlations. However, widely used methods such as uniform experience replay
(UER) and prioritized experience replay (PER) have been shown to have
sub-optimal convergence and high seed sensitivity respectively. To address
these issues, we propose a novel approach called IntrospectiveExperience Replay
(IER) that selectively samples batches of data points prior to surprising
events. Our method builds upon the theoretically sound reverse experience
replay (RER) technique, which has been shown to reduce bias in the output of
Q-learning-type algorithms with linear function approximation. However, this
approach is not always practical or reliable when using neural function
approximation. Through empirical evaluations, we demonstrate that IER with
neural function approximation yields reliable and superior performance compared
toUER, PER, and hindsight experience replay (HER) across most tasks.

Machine Learning