Thanks! I haven’t thought about the i2h and h2o thing.

In general, you don’t want to call  loss.backward()  every time step of an RNN. You want to compute a few timesteps and the compute the loss across all the outputs

I know that this implementation is not optimal, in fact i tried the same but with batches (and computing the loss at each mini batch), but i tried to keep it as simple as i could to see if the gradients changed (which have not).
I will try it again, implementing batches and see if it works.