Solved by jpeg729 in post #4 
              
              
                To convert (3, N, hidden_size) to (N, *, in_feature) use .permute(1,0,2) 
Permute interchanges the dimensions. 
Besides why not make the first layer of your decoder take input of shape (3, N, 40)?