I see, I am gonna manual_seed the model and then I will see how it behaves. However, since I see things are okay(ish) with resnet18/34 I may stick to transfer learning. Specifically given how small the data set actually is.