a_d:

There is no change in losses in FP16.


In your code snippet you are not recreating an optimizer for the AMP run, so the model won’t be updated.



 a_d:

One should expect the same behavior right? like losses might be different but it should decrease.


Generally that’s the assumption. However, from my experience this assumption is valid, if the FP32 training is “healthy”, i.e. if the loss/activations are not blowing up (i.e. the model diverges), as the AMP training might not be able to recover from this state.
After adding the missing optimizer, note that both runs indeed “change”, but the overall training might not be well defined.
E.g. in some runs the FP32 loss seems to get negative values and explodes afterwards:
tensor(-30.6151, device='cuda:0', grad_fn=<AddBackward0>)
tensor(-35.7750, device='cuda:0', grad_fn=<AddBackward0>)
tensor(-42.5553, device='cuda:0', grad_fn=<AddBackward0>)
tensor(-52.2006, device='cuda:0', grad_fn=<AddBackward0>)
tensor(-67.1675, device='cuda:0', grad_fn=<AddBackward0>)
tensor(-93.4945, device='cuda:0', grad_fn=<AddBackward0>)
tensor(-150.9680, device='cuda:0', grad_fn=<AddBackward0>)
tensor(-356.5783, device='cuda:0', grad_fn=<AddBackward0>)
tensor(2775.4045, device='cuda:0', grad_fn=<AddBackward0>)
tensor(369.3389, device='cuda:0', grad_fn=<AddBackward0>)

in others both losses seem to be reduced:
FP32
tensor(0.6024, device='cuda:0', grad_fn=<AddBackward0>)
tensor(0.6009, device='cuda:0', grad_fn=<AddBackward0>)
tensor(0.5995, device='cuda:0', grad_fn=<AddBackward0>)
tensor(0.5981, device='cuda:0', grad_fn=<AddBackward0>)
tensor(0.5967, device='cuda:0', grad_fn=<AddBackward0>)
tensor(0.5953, device='cuda:0', grad_fn=<AddBackward0>)
tensor(0.5939, device='cuda:0', grad_fn=<AddBackward0>)
tensor(0.5924, device='cuda:0', grad_fn=<AddBackward0>)
tensor(0.5910, device='cuda:0', grad_fn=<AddBackward0>)
tensor(0.5896, device='cuda:0', grad_fn=<AddBackward0>)
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-==-==-=-==-==-=-==-==-=-=-==-
AMP
tensor(0.5286, device='cuda:0', grad_fn=<AddBackward0>)
tensor(0.5277, device='cuda:0', grad_fn=<AddBackward0>)
tensor(0.5268, device='cuda:0', grad_fn=<AddBackward0>)
tensor(0.5258, device='cuda:0', grad_fn=<AddBackward0>)
tensor(0.5249, device='cuda:0', grad_fn=<AddBackward0>)
tensor(0.5239, device='cuda:0', grad_fn=<AddBackward0>)
tensor(0.5230, device='cuda:0', grad_fn=<AddBackward0>)
tensor(0.5220, device='cuda:0', grad_fn=<AddBackward0>)
tensor(0.5211, device='cuda:0', grad_fn=<AddBackward0>)
tensor(0.5202, device='cuda:0', grad_fn=<AddBackward0>)

while others break with a negative loss again:
FP32
tensor(5.8274, device='cuda:0', grad_fn=<AddBackward0>)
tensor(5.6770, device='cuda:0', grad_fn=<AddBackward0>)
tensor(5.5385, device='cuda:0', grad_fn=<AddBackward0>)
tensor(5.4073, device='cuda:0', grad_fn=<AddBackward0>)
tensor(5.2816, device='cuda:0', grad_fn=<AddBackward0>)
tensor(5.1607, device='cuda:0', grad_fn=<AddBackward0>)
tensor(5.0440, device='cuda:0', grad_fn=<AddBackward0>)
tensor(4.9311, device='cuda:0', grad_fn=<AddBackward0>)
tensor(4.8220, device='cuda:0', grad_fn=<AddBackward0>)
tensor(4.7163, device='cuda:0', grad_fn=<AddBackward0>)
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-==-==-=-==-==-=-==-==-=-=-==-
AMP
tensor(-9.6253, device='cuda:0', grad_fn=<AddBackward0>)
tensor(-9.6253, device='cuda:0', grad_fn=<AddBackward0>)
tensor(-9.6253, device='cuda:0', grad_fn=<AddBackward0>)
tensor(-9.6253, device='cuda:0', grad_fn=<AddBackward0>)
tensor(-9.6253, device='cuda:0', grad_fn=<AddBackward0>)
tensor(-9.6253, device='cuda:0', grad_fn=<AddBackward0>)
tensor(-9.6253, device='cuda:0', grad_fn=<AddBackward0>)
tensor(-9.6253, device='cuda:0', grad_fn=<AddBackward0>)
tensor(-9.6253, device='cuda:0', grad_fn=<AddBackward0>)
tensor(-9.6253, device='cuda:0', grad_fn=<AddBackward0>)

Is a negative loss expected in your case?