I am sorry I misunderstood your question at first.
But with the information given I can’t say anything more.
I also tried to recreate your problem but couldn’t. Everything works fine on my end.
There is no noticeably large difference in loss between using model.eval() or model.train() after loading.