Thank you, your answer is much appreciated.
I have tried bidirectional and also increasing complexity otherwise (more layers, played with hidden and embedding dimensions, …) but nothing seemed to make a real difference.
One of the things that is surprising me the most about this learning experience, is how little regularization actually helped. I was under the impression it would be way more effective. It definitely leads to the train-loss decreasing slower but sadly it doesn’t lead to the validation-loss decreasing further to nearly the same extent.
I will definitely look at BLEU score and try to find a bigger dataset, thank you!