I really appreciate all your suggestions!



 vdw:

Can you overfit the data on a (very) small training dataset, i.e., can you achieve a training loss of 0 if you train over only, say, 50 sentence pairs?


Yes, after about 200 epochs the loss is 0.0002. If I kept going it would probably get to 0 but at that point training is pretty slow.



 vdw:

I don’t have any solid argument, but a dropout after the the embedding layer always looks a bit weird to me.


I don’t really have a solid argument for that either. I saw someone doing it and gave it a shot. I don’t really get why it would be a good idea either though.



 vdw:

Not sure how your sequences look like, but I assume you properly add and tokens to your sequences before adding the padding token.


I assumed the padding didn’t matter since the loss-function is ignoring it anyway. You’re right though, I could try it without padding.



 vdw:

From your code it seems you use batches of size > 1 and padding. I’m actually not quite sure how this works for the decoder part. The official Seq2Seq tutorial of PyTorch uses batches of size 1 to avoid this issue; sacrificing performance of course. Anyway, for testing, I would try to train using batches of size 1 to avoid any kind of padding for the input and target sequence.


I’m not sure if there’s something missing in that sentence or if I’m just not getting it. Could you elaborate on that?