The training loss is different, but is it qualitatively different? If it’s not qualitatively different (i.e. it’s not better or worse in any stastically significant way), I’d say it’s expected (read below).
The most obvious ways why the training loss is diverging is that you are not saving / restoring the random seed for the DataLoader RNG (I see you have RandomAffine / RandomHorizontalFlip).