I don’t know much about RNNs, but your model looks fine.
After a second glance at your code, did you try removing the retain_graph=True from your backward pass? It is only necessary if you are doing double backprop or some other tricks.
I’m not sure if that’s the reason or not.
If the issue still persists, could you try creating a small example (~100 lines) that reproduces your problem?
Thanks!