Lei_Carl:

I called the optim.step() again and it resulted in the accuracy loss.


As described in the docs 3 you are not supposed to call optimizer.step(), so use the described approach.
I don’t know where the slowdown is coming from, but in case you are using CNNs you could trry to use torch.backends.cudnn.benchmark = True and see if it would speedup your workload again.