Here is an update: I ran several more times and found that if I comment the “loss.backward()” line, the speed will be normal. Given that I have to keep this line for sure, how to make the training speed normal?
I believe it must be the data somehow overflowed, so pytorch have to clear the cache at every iteration. But why and how it caused this problem?