I’ve just seen that you only stack two of the encoder layers what might be too few. Talking about your hidden dimension, shouldn’t it be hid_dim = embed_dim as using different dimensions would complicate it. You could also try 300d for instance.