Hey…looks like I found the problem! So, when I was investigating further how to covert from Keras to PyTorch, I tried to keep things as much as possible the same (like initialisation schemes and such), so I changed the Adadelta default values to correspond to the ones in Keras. In Keras, eps = 1e-7 and it turns out that the model learns as it did before if I keep the epsilon value at the default in PyTorch, which is 1e-6. Nevertheless, this is a weird behaviour…if anyone has an explanation please share.
Thank you!