Thank you! Indeed, pushing the model to GPU before loading the state of optimiser solved the problem. Though, I would say that the error message should pop up upon loading the state dict, not on step(). Thanks again @ptrblck and @Upgrade_Yourself!