It’s hard to tell, where the accuracy drop might be coming from and I would recommend to load a specific image sample during training and validation and compare the output of these runs.
If your model doesn’t give the same output (in eval() mode), than something might be different in the data loading or preprocessing.