Thanks for your reply. I did what you said. Before I can test your suggestion, now I’m facing some other issue.
Without fp16, I’m able to train with batch size 128.
After amp.initialize() is called, I am getting an OOM error with the same batch size which is not ideal since RAM usage should decrease. A similar issue has been raised here 1.