Thanks a lot for replying @ptrblck . Sorry I did not notice that I missed the zero_grad.  I missed it in my original model too and freaked out . On my single data point, the model converges decently in both modes.
But even when not using zero_grad, why did FP32 loss decrease but FP16 remain constant(even though model weights change). The loss should at least vary right(since the weights are not the same after optim.step())?