Thanks for you reply. It works well now. I called the optim.step() again and it resulted in the accuracy loss.
But another problem is that: on the gpu with Tensor core, both native amp and apex(O1 mode) have a larget increase of speed. But on the 1080ti (which don’t have Tensor core), the native amp is 2x slower than apex(O1 mode). When I change the apex mode to O2, it also become 2x slower. It can be confirmed that the speed of native amp and apex(O2 mode) decreases obviously due to the fp16 power bottleneck of 1080Ti. Is there any way to change the mode of native amp to remove some fp16 cast. The O1 mode works well on 1080ti(the speed didn’t improve, but the memory footprint was much lower). I’m dying to use the native amp since it requires much fewer change of the origin code.