To use multiple inputs, you can concatenate them on last dimension (torch.cat).
Note that you may not need x/y role split in the middle of the network, unless you have residual architecture, multiple outputs/losses or something like that - your second hidden layer can produce hidden x representation that is fed to rnn, and Y can be created later.