The OP is actually trying to help me out with this problem.  However, the post is actually backwards ( i think).  The objective is to create a neural network as pictured in the original image.  I was able to code the portion of the network that is outlined in red; however, I would like to add the portion in blue.  My main problem is that I am so new to pytorch that I don’t know how to actually do it.  Hence, me recruiting help, and finding it here ;).
To the best of my ability I created a model that utilizes word embeddings for an input and number sequences as output.  The number sequences are coded in a manner in which they represent the word that corresponds to the embedding.  The input and output are connected by two hidden layers, one of which is recurrent.
The language utilized in the image that reflects the current network I have created follows this:
Embeddings <-> recurrent <-> hidden <-> arpabet_out
What my goal is now to add the remaining structure of the intended network which would look like this:
Embeddings <-> hidden <-> Arpabet_in AND
arpaet_in <-> hidden <-> recurrent <-> arpabet_out
Please let me know if I can provide any more information.  Should it be easier to look at all the code; I have this on my original post in which you linked.