I gave up on trying to improve that, I double-checked almost everything and it is still with the same suboptimal results when compared to TensorFlow. I really think that Pytorch should have more tests on the convergence of different networks, because it seems that something very subtle is happening, and it is almost impossible to find the issue, even on very simple networks.