I’m a pytorch noob so take my answer with a grain of salt. But I am working on a similar problem and have just got it working. From your code segment I cannot tell if you have implemented the weight tie in the forward function or not, but that was the problem for me. I originally tied the weights in the init but this did not force the weights to update together, so during learning time, they drifted apart. Instead now I have used functional methods in the decoder section, and directly connected the weights in that forward function… so far this seems to work.