  The primacy bias in deep reinforcement learning (DRL), which refers to the
agent's tendency to overfit early data and lose the ability to learn from new
data, can significantly decrease the performance of DRL algorithms. Previous
studies have shown that employing simple techniques, such as resetting the
agent's parameters, can substantially alleviate the primacy bias. However, we
observe that resetting the agent's parameters harms its performance in the
context of model-based reinforcement learning (MBRL). In fact, on further
investigation, we find that the primacy bias in MBRL differs from that in
model-free RL. In this work, we focus on investigating the primacy bias in MBRL
and propose world model resetting, which works in MBRL. We apply our method to
two different MBRL algorithms, MBPO and DreamerV2. We validate the
effectiveness of our method on multiple continuous control tasks on MuJoCo and
DeepMind Control Suite, as well as discrete control tasks on Atari 100k
benchmark. The results show that world model resetting can significantly
alleviate the primacy bias in model-based setting and improve algorithm's
performance. We also give a guide on how to perform world model resetting
effectively.

Machine Learning;Artificial Intelligence