I am training the network on google colab GPU. If I interrupt the training and continue it by loading saved weights, I do not see this problem.
But, if I factory reset the runtime. Reload the weights, then I can see jump in training loss.
I am also setting the same global random seed in my colab notebook.