Were you able to exactly reproduce the same model parameters for your runs?
I.e. did you compare the state_dicts before running the validation loop?
Even if you are seeding and getting the same data samples for your runs, the result might still differ, e.g. due to cudnn as described in the Reproducibility docs 30.