@sid, yes, that is what I am doing as well… Thank you @ptrblck, the implementation of amp into pytorch works really well. It almost seems faster… I’m not sure if the error mentioned above will go away, but again, pytorch’s amp new feature is nice to have.