the issue being when eps is set to be 1e-8, as is the default, and used with autocast, the network’s loss will inevitably be Null after some epoch. However, increasing eps to higher value seems to make it go away, yet when compare to training not done on fp16, the convergence is much slower.