If I understand the issue correctly, you get a constant validation loss for the complete validation dataset after restoring the model, while the loss for the same setup changes before storing the model?
If so, could you post the model definition, so that we could have a look?