I don’t know what might be causing this effect as I haven’t seen or ran any experiments by frequently changing the precision.



 sdfadsfa:

Also I’m not sure how to set GradScaler’s enabled=False dynamically. Isn’t that started before the training, and you can’t change it afterwards?


Yes, usually you could either enable or disable amp in your script which can be done with the enabled arguments. If you want to dynamically switch the precision, you could use if conditions to pick the appropriate code path.