I can’t see anything obvious that looks suspicious, so here just some comments as food for thoughts. By the way, I appreciate that you keep track your dimensions in the comments :). This always helps a lot, I think.


Can you overfit the data on a (very) small training dataset, i.e., can you achieve a training loss of 0 if you train over only, say, 50 sentence pairs?


I don’t have any solid argument, but a dropout after the the embedding layer always looks a bit weird to me.


From your code it seems you use batches of size > 1 and padding. I’m actually not quite sure how this works for the decoder part. The official Seq2Seq tutorial of PyTorch uses batches of size 1 to avoid this issue; sacrificing performance of course. Anyway, for testing, I would try to train using batches of size 1 to avoid any kind of padding for the input and target sequence.


Not sure how your sequences look like, but I assume you properly add  and  tokens to your sequences before adding the padding token.


Lastly, machine translation is not a trivial task just because big companies might be able to do it almost flawlessly by now. So it’s not obvious to say what kind of loss can be expected in the end given basic model. Hence, my suggestion to see if you can overfit the model over a very small dataset.