Are you using Adam optimizer? If yes then I’d suggest you to try setting it’s eps to bigger value like 1e-6 or 1e-4 or even bigger if required, (where it’s default value is 1e-8, as mentioned here 16).
Because generally Adam optimizer won’t work well with autocast or half precision.