Not really any solution – I cannot see any obvious issues in your code – just some ideas you might want to consider. It essentially aims to simplify to a more tried and tested decoder architecture:


I’ve never seen a linear layer between an embedding and an RNN layer. Not sure what this means semantically, but I would skip that one for a start.


It don’t think it makes sense that the decoder is bidirectional. Since you feed it one token/word step by step, the output should be the same anyway. I would remove the stacking to remove any potential point of error.


I would start with the basic decoder (without attention; not relevant for you anyway) of the basis Seq2Seq tutorial 3. Again, just because it’s tried and tested…including by myself :).


To accommodate the Seq2Seq tutorial, I would also work with batches of size 1. You seem to use larger batches, but that’s not obivous to do for a decoder. I only do it when I know that that my target sequences in a batch have the same lengths (e.g., for an autoencoder).


In short, I would start with the most basic architecture/setup using tested code (snippets) as far as possible (and batch_size=1) to get it working (= easy overfitting on a small dataset). Once that seems fine, I add complexity to improve performance and/or accuracy – or to modify the basic model to my specific use case.