I haven’t noticed any problems. Try SGD optimizer with big initial lr and ExponentialLR scheduler (there is no scheduler.step() in your code btw). Or optimizing without mini-batches (Rprop or LBFGS optimizers).