If the randomness is introduced only after resuming the training, it could be the reason.
However, if the input tensors use the same data augmentation / transformation it shouldn’t increase the loss unexpectedly.
I would recommend to check the model with a static input tensor, e.g. torch.ones before saving and after loading the model to make sure the output is the same (call model.eval() to disable e.g. dropout layers). If these outputs don’t match (up to floating point precision), the model loading itself seems to fail.