I managed to fix this by clipping the gradients with                             torch.nn.utils.clip_grad_norm_(self.actor_critic.parameters(), self.cfg.max_grad_norm)
Unfortunately this slows down the code, and I didn’t have to do it before. Weird.