I tried with the changes you suggested. It is still the same result. Also I can visually notice the time difference because pytorch code is 4-5 times slower than tensorflow code.
My pytorch version is 1.4.0
Cuda version is 10.1.243
GPU is Geforce GTX 1080 ti