The answer to this question depends on how many layers and directions your decoder has. If you did not come across this earlier, I highly recommend @bentrevett’s (a huge shoutout to him assuming he is the same as the owner of this repo) pytorch-seq2seq tutorials 2 where he walks you through all the shape transformations inside the encoder and decoder.