@ptrblck and @albanD It would be awesome if you can share some ideas about this.
One more update: I’m using the procgen environment, for the Coinrun, I can only train 1.1M frames. But I trained the same model on the Maze environment, it reached 5M frames without error