I used the same setup(TITAN RTX GPU and linux etc), only changed torch version.
Python3.6 with pytorch version 1.2 → python3.6 with pytorch version 1.6.
The dramatic change occurs when I update the weights of the network during training.
For example, during training, the first iteration prediction and loss are the same for both versions but when I use optimizer.step(), second iteration prediction, and loss are so different.
So in PyTorch 1.6 version, eventually results in so bad performance.
I looked at the documentation of the optimizer.step(), but it seems there is no difference between pytorch1.2 and pytorch1.6 versions.
This is the code for my optimizers that trains the network.

image780×273 11.7 KB