First off thank you for the great answer. I am very confused as well because not only are the results not garbled and coherent, but they are exact sentences from other examples. I tried to overtrain the model, and it works on a tiny dataset of 5 examples but when I bring it up to 10 examples the same effect starts to occur.
As for your first question, I want to concatenate the hidden states together from both directions and pass them through a linear to scale them down to one hidden state size to pass to the decoder.
For the second question, I don’t need the encoder output and I don’t use it. The decoder_input is set initially to be the start token * the batch size, though your way is more elegant. I’m pretty sure it does the same thing though.
I am in the process of going through your example, will let you know how it goes.
Thanks for all the help!