Solved by tom in post #4 
              
              
                Yeah, you have got all the ingredients there, but compare your reshaping with your rnn output and the linear in size. So you need to multiply the self.hidden_size by self.direction in the reshape.