but it seems you are initializing two to create a bidirectional layer?

Yes, Keras has also a bidirectional combo layer but you can also set them up separately and set a property ‘backward’ on one of it so it flips the input within the layer.



 toho:

optimizer.zero_grad() loss.backward() optimizer.step()


I got the order of this triple wrong 
I zeroed the gradients too early.
Thanks anyway