I found the issue. I will mention it here for anyone who come across this issue later.
I’m using Adam where the default eps in Keras is 1e-7, whereas the default value in Pytorch is 1e-8… Also the default Conv2d initialization is different from Keras.
After fixing both eps and initialization. It converged nicely like what it did in Keras.