Thanks. It’s caused by denormalized floats.
Both set_flush_denormal and setting all parameters close to zero to exactly zero can solve this problem.

torch.set_flush_denormal(True)
Set all parameters close to zero to exactly zero

for param in model.parameters():
    param.data = torch.where(torch.abs(param) < eps, torch.zeros_like(param), param)