If you can isolate the different loss curves using the same data augmentation and the same data, I would think the difference might come from skewed batchnorm running estimates.
Could you set track_running_stats=False for all batchnorm layers and rerun the experiments?