I didn’t call half() on any tensors or models. The code is running fine without using apex at all, or using apex with opt_level=‘O1’
Please note that by using apex, I changed from torch’s Adam optimizer to the FusedAdam in apex.optimizers. I will test if by using torch’s adam the error will go away.
I am attaching a screenshot of the error message (sorry I have to blackout the user information). The exact time this error happen seems to be random: some times it happens after 100 epoch.
Screenshot from 2020-05-16 18-05-181108×762 134 KB