Besides disabling data augmentation, do you still have the batchnorm layers in your model and call model.eval()?
Which loss curve do you get, when you pass the training set (with data augmentation) to the model after calling model.eval()?